2024-09-13T11:20:11.993 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T11:20:12.013 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:20:12.110 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421 branch: wip-aclamk-cbt-combined description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/pggrow_host thrashosds-health workloads/dedup-io-mixed} email: akupczyk@redhat.com first_in_suite: false job_id: '7903421' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore odsync write: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd max backfills: 2 osd max markdown count: 1000 osd max scrubs: 3 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 0.5 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - \(POOL_APP_NOT_ENABLED\) sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 min_in: 4 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 5394 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 582/12160 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHXVysjpVe/7grEmhcU1oTH1zIv/vx+Pgw5HFraDCdmShzubyuI0kMjUsVH/xajsaE+f8dBouuglNZGG0D3pqY= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAB9yTrwh3eB8qVf+11IbFEQ+tBTYi2cDpSWXY/dlcYdV6hrHY3F6IlLCozJZfBEo+tYtKJ34pvZYm9rc2VwqUs= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCK+8vNd70wRg3y5L81N0Z/3GBvj4rvzoAse31fmhzYvm4f9RThrnjCTLU3CjdJJYpQnd4/CofalBb9uNPqiMYw= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3fsQAGRtm6fT60TA+6AUzgeo2jfbutPDpZCruJccQwNekzsnOHIeDYNuls2/YL/B2kVQ+jJIY9657AKXZXSPY= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 2 chance_pgpnum_fix: 1 min_in: 2 thrash_hosts: true timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 dedup_chunk_algo: fastcdc dedup_chunk_size: '131072' enable_dedup: true low_tier_pool: low_tier objects: 50 op_weights: read: 100 set_chunk: 30 tier_evict: 10 tier_flush: 5 tier_promote: 10 write: 50 ops: 1500 set_chunk: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:37:56 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T11:20:12.110 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T11:20:12.111 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T11:20:12.111 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T11:20:12.112 INFO:teuthology.task.internal:Checking packages... 2024-09-13T11:20:12.133 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T11:20:12.133 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T11:20:12.133 INFO:teuthology.packaging:ref: None 2024-09-13T11:20:12.133 INFO:teuthology.packaging:tag: None 2024-09-13T11:20:12.133 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T11:20:12.133 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:20:12.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T11:20:12.284 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T11:20:12.285 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T11:20:12.292 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T11:20:12.292 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T11:20:12.312 INFO:teuthology.task.internal:Saving configuration 2024-09-13T11:20:12.328 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T11:20:12.335 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T11:20:12.360 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:14:37.592269', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHXVysjpVe/7grEmhcU1oTH1zIv/vx+Pgw5HFraDCdmShzubyuI0kMjUsVH/xajsaE+f8dBouuglNZGG0D3pqY='} 2024-09-13T11:20:12.383 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:14:37.584485', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAB9yTrwh3eB8qVf+11IbFEQ+tBTYi2cDpSWXY/dlcYdV6hrHY3F6IlLCozJZfBEo+tYtKJ34pvZYm9rc2VwqUs='} 2024-09-13T11:20:12.403 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:14:37.589084', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCK+8vNd70wRg3y5L81N0Z/3GBvj4rvzoAse31fmhzYvm4f9RThrnjCTLU3CjdJJYpQnd4/CofalBb9uNPqiMYw='} 2024-09-13T11:20:12.428 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 11:14:37.586804', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3fsQAGRtm6fT60TA+6AUzgeo2jfbutPDpZCruJccQwNekzsnOHIeDYNuls2/YL/B2kVQ+jJIY9657AKXZXSPY='} 2024-09-13T11:20:12.428 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T11:20:12.435 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-13T11:20:12.435 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-13T11:20:12.435 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-13T11:20:12.435 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-13T11:20:12.435 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T11:20:12.541 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd87ceb3be0>, signals=[15]) 2024-09-13T11:20:12.541 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T11:20:12.547 INFO:teuthology.task.internal:Opening connections... 2024-09-13T11:20:12.547 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:20:12.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:20:12.627 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:20:12.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:20:12.701 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:20:12.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:20:12.778 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:20:12.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:20:12.853 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T11:20:12.859 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-09-13T11:20:12.878 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-09-13T11:20:12.878 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-09-13T11:20:12.936 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-09-13T11:20:12.936 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-09-13T11:20:12.936 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-09-13T11:20:12.936 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:20:12.936 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-09-13T11:20:12.936 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:20:12.936 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:20:12.937 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-09-13T11:20:12.937 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:20:12.937 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:20:12.937 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:20:12.937 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:20:12.937 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:20:12.937 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:20:12.938 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-09-13T11:20:12.956 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-09-13T11:20:12.974 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-09-13T11:20:12.974 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-09-13T11:20:13.031 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-09-13T11:20:13.031 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-09-13T11:20:13.031 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-09-13T11:20:13.031 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:20:13.031 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-09-13T11:20:13.031 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:20:13.032 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:20:13.032 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-09-13T11:20:13.056 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-09-13T11:20:13.074 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-09-13T11:20:13.075 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:20:13.133 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:20:13.134 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:20:13.134 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:20:13.134 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:20:13.134 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-09-13T11:20:13.156 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-09-13T11:20:13.175 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-09-13T11:20:13.175 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T11:20:13.233 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-09-13T11:20:13.234 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-09-13T11:20:13.234 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T11:20:13.234 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-09-13T11:20:13.234 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T11:20:13.234 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T11:20:13.234 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T11:20:13.234 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-09-13T11:20:13.256 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T11:20:13.265 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T11:20:13.293 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T11:20:13.294 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:20:13.296 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:20:13.298 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:20:13.300 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-09-13T11:20:13.317 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T11:20:13.323 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T11:20:13.323 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:20:13.353 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:20:13.356 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:20:13.358 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-09-13T11:20:13.467 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T11:20:13.558 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T11:20:13.571 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T11:20:13.572 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T11:20:13.572 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:20:13.572 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:20:13.573 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:20:13.573 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T11:20:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:13.591 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-09-13T11:20:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:20:13.593 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-13T11:20:13.593 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-13T11:20:13.594 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-09-13T11:20:13.648 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:20:13.648 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-503.el9.x86_64 2024-09-13T11:20:13.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-09-13T11:20:13.651 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:20:13.651 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.14.0-503.el9.x86_64 2024-09-13T11:20:13.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2024-09-13T11:20:13.652 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:20:13.653 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-503.el9.x86_64 2024-09-13T11:20:13.653 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.14.0-503.el9.x86_64 2024-09-13T11:20:13.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2024-09-13T11:20:13.653 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-503.el9.x86_64 2024-09-13T11:20:13.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-09-13T11:20:16.103 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-13T11:20:16.223 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-13T11:20:16.721 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 8.2 MB 00:01 2024-09-13T11:20:17.171 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-09-13T11:20:18.007 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-13T11:20:18.797 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-13T11:20:18.956 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-13T11:20:19.962 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 20 MB 00:02 2024-09-13T11:20:23.191 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.5 MB 00:01 2024-09-13T11:20:23.215 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-13T11:20:24.065 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-13T11:20:24.204 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-13T11:20:24.692 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 116 kB/s | 19 kB 00:00 2024-09-13T11:20:24.783 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 19 kB 00:00 2024-09-13T11:20:25.496 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-13T11:20:25.540 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 19 kB 00:00 2024-09-13T11:20:25.789 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 19 kB 00:00 2024-09-13T11:20:25.878 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-13T11:20:26.536 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-13T11:20:26.643 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-13T11:20:31.161 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T11:20:31.276 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-09-13T11:20:31.934 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T11:20:32.057 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-13T11:20:33.243 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:20:33.243 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:20:33.257 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:20:33.258 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:20:33.336 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:20:33.341 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:20:33.347 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:20:33.348 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:20:33.348 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:20:33.348 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:20:33.348 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:20:33.348 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-13T11:20:33.348 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:20:33.349 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 79 M 2024-09-13T11:20:33.349 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 126 M 2024-09-13T11:20:33.349 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:20:33.352 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:20:33.352 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-13T11:20:33.352 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:20:33.352 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:20:33.352 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:20:33.352 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:20:33.352 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:20:33.353 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:20:33.353 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:20:33.353 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:20:33.353 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:20:33.353 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:20:33.353 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-09-13T11:20:33.353 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:20:33.354 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 79 M 2024-09-13T11:20:33.354 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 126 M 2024-09-13T11:20:33.354 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:20:33.513 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 201 kB/s | 23 kB 00:00 2024-09-13T11:20:33.669 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 76 kB/s | 23 kB 00:00 2024-09-13T11:20:33.896 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-13T11:20:33.982 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:20:33.982 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:20:34.087 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T11:20:34.087 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T11:20:34.090 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:20:34.101 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:20:34.102 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:20:34.102 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:20:34.102 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:20:34.102 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2024-09-13T11:20:34.102 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:20:34.102 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 79 M 2024-09-13T11:20:34.102 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 126 M 2024-09-13T11:20:34.103 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:20:34.191 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T11:20:34.204 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 79 M 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 126 M 2024-09-13T11:20:34.205 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:20:34.236 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 202 kB/s | 23 kB 00:00 2024-09-13T11:20:34.288 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 37 MB/s | 29 MB 00:00 2024-09-13T11:20:34.309 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 277 kB/s | 23 kB 00:00 2024-09-13T11:20:34.597 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-13T11:20:34.597 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:20:34.597 INFO:teuthology.orchestra.run.smithi038.stdout:Total 63 MB/s | 79 MB 00:01 2024-09-13T11:20:35.306 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:20:35.442 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-13T11:20:35.703 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:20:35.703 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:20:36.067 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 16 MB/s | 29 MB 00:01 2024-09-13T11:20:36.665 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:20:36.666 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:20:36.851 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 13 MB/s | 35 MB 00:02 2024-09-13T11:20:36.851 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:20:36.851 INFO:teuthology.orchestra.run.smithi154.stdout:Total 30 MB/s | 79 MB 00:02 2024-09-13T11:20:37.537 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:20:37.895 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:20:37.895 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:20:38.176 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:04 2024-09-13T11:20:38.685 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.6 MB/s | 35 MB 00:04 2024-09-13T11:20:38.854 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:20:38.855 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:20:38.944 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.1 MB/s | 29 MB 00:04 2024-09-13T11:20:38.944 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:20:38.944 INFO:teuthology.orchestra.run.smithi118.stdout:Total 16 MB/s | 79 MB 00:04 2024-09-13T11:20:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:20:39.629 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:20:39.918 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:20:39.993 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:20:39.993 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:20:40.009 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:40.927 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.6 MB/s | 35 MB 00:07 2024-09-13T11:20:40.967 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:20:40.967 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:20:41.323 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:20:41.997 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:20:42.077 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:42.144 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 3.4 MB/s | 29 MB 00:08 2024-09-13T11:20:42.802 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:42.928 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:43.455 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:20:44.129 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:20:44.210 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:44.892 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:45.032 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:45.235 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 1.3 MB/s | 16 MB 00:11 2024-09-13T11:20:45.236 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:20:45.236 INFO:teuthology.orchestra.run.smithi204.stdout:Total 6.6 MB/s | 79 MB 00:11 2024-09-13T11:20:45.917 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:20:46.268 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:20:46.268 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:20:47.012 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:47.149 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:47.266 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:20:47.267 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:20:48.050 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:48.095 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:20:49.728 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:20:49.885 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:49.915 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:20:50.361 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:20:50.520 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:51.988 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:52.032 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:20:53.238 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:20:53.323 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:20:53.459 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:20:56.874 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:20:58.324 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:20:58.442 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:03.272 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:32.478 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:32.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:21:32.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:21:32.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:21:32.963 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:33.004 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:33.329 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:33.329 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:21:33.330 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:21:33.330 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:33.330 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:33.330 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:21:33.330 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:33.330 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:21:33.330 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:21:33.377 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:33.377 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:21:33.377 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:21:33.378 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:21:33.706 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:33.796 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:33.796 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:33.796 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:21:33.797 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:21:33.797 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:21:33.797 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T11:21:33.797 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:21:33.797 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:33.855 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:33.856 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:33.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-09-13T11:21:33.887 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-09-13T11:21:33.887 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.06 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 82.el9 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Group : Unspecified 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 8274670 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : builder@centos.org 2024-09-13T11:21:33.888 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:21:33.889 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:21:33.890 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:21:33.890 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:21:33.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:21:34.286 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:34.286 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:21:34.286 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:21:34.287 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:34.287 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:34.287 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:21:34.287 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:34.287 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:21:34.287 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:21:34.641 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:34.679 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-09-13T11:21:34.732 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:34.733 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:34.733 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:21:34.733 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:21:34.733 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:21:34.733 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-13T11:21:34.733 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:21:34.733 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:34.791 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:34.792 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:34.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.06 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 82.el9 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Group : Unspecified 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 8274670 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2024-09-13T11:21:34.831 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : builder@centos.org 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:21:34.832 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2024-09-13T11:21:34.833 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:21:34.833 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:21:34.834 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:21:34.834 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:21:34.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:21:35.480 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-09-13T11:21:35.776 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:35.816 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:36.023 INFO:teuthology.orchestra.run.smithi038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:21:36.052 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-09-13T11:21:36.057 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-09-13T11:21:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.ZChOjQZiYp 2024-09-13T11:21:36.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZChOjQZiYp 2024-09-13T11:21:36.181 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:36.181 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:21:36.181 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:21:36.182 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:21:36.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.ZChOjQZiYp 2024-09-13T11:21:36.324 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.ZChOjQZiYp is 6KB 2024-09-13T11:21:36.374 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.ZChOjQZiYp 2024-09-13T11:21:36.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:21:36.452 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:21:36.452 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:21:36.452 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:21:36.452 INFO:teuthology.orchestra.run.smithi038.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:21:36.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:21:36.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-09-13T11:21:36.803 INFO:teuthology.orchestra.run.smithi154.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:21:36.822 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-09-13T11:21:36.829 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-13T11:21:36.863 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.id6qhBZQTA 2024-09-13T11:21:36.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.id6qhBZQTA 2024-09-13T11:21:36.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.id6qhBZQTA 2024-09-13T11:21:37.032 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:21:37.033 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:21:37.082 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.id6qhBZQTA is 6KB 2024-09-13T11:21:37.131 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.id6qhBZQTA 2024-09-13T11:21:37.146 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:21:37.208 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:21:37.209 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:21:37.209 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:21:37.209 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:21:37.210 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:21:37.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-09-13T11:21:37.387 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:37.468 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:37.469 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:37.469 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:21:37.469 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:21:37.469 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:21:37.469 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T11:21:37.469 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:21:37.470 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:37.529 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:37.529 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:37.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-09-13T11:21:37.566 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-09-13T11:21:37.566 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 82.el9 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8274670 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2024-09-13T11:21:37.567 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:21:37.568 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:21:37.569 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:21:37.569 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:21:37.569 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:21:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-09-13T11:21:39.684 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:21:39.715 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-09-13T11:21:39.718 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-09-13T11:21:39.734 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.55LWClq04z 2024-09-13T11:21:39.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.55LWClq04z 2024-09-13T11:21:39.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.55LWClq04z 2024-09-13T11:21:39.938 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.55LWClq04z is 6KB 2024-09-13T11:21:39.986 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.55LWClq04z 2024-09-13T11:21:40.000 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:21:40.063 INFO:teuthology.orchestra.run.smithi118.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:21:40.063 INFO:teuthology.orchestra.run.smithi118.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:21:40.064 INFO:teuthology.orchestra.run.smithi118.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:21:40.064 INFO:teuthology.orchestra.run.smithi118.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:21:40.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:21:40.211 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-09-13T11:21:41.716 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:41.756 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T11:21:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T11:21:42.123 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T11:21:42.949 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T11:21:42.949 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:21:42.949 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:21:42.950 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:42.950 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:42.950 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T11:21:42.950 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T11:21:42.950 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:21:42.950 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:21:43.276 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:43.399 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:43.399 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:43.399 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T11:21:43.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T11:21:43.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T11:21:43.400 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-13T11:21:43.400 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T11:21:43.400 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T11:21:43.459 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T11:21:43.459 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T11:21:43.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-09-13T11:21:43.492 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-09-13T11:21:43.492 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-09-13T11:21:43.492 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-09-13T11:21:43.492 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 82.el9 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8274670 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-09-13T11:21:43.493 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-09-13T11:21:43.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:21:43.494 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T11:21:43.494 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T11:21:43.494 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T11:21:43.494 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-09-13T11:21:43.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:21:43.494 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T11:21:43.495 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T11:21:43.495 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-505.el9.x86_64 2024-09-13T11:21:43.495 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T11:21:44.092 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-09-13T11:21:45.485 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T11:21:45.515 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-09-13T11:21:45.517 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-09-13T11:21:45.550 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.d3FCggFHy4 2024-09-13T11:21:45.550 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.d3FCggFHy4 2024-09-13T11:21:45.617 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.d3FCggFHy4 2024-09-13T11:21:45.763 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.d3FCggFHy4 is 6KB 2024-09-13T11:21:45.813 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.d3FCggFHy4 2024-09-13T11:21:45.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T11:21:45.898 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T11:21:45.898 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T11:21:45.898 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T11:21:45.898 INFO:teuthology.orchestra.run.smithi204.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T11:21:45.899 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T11:21:46.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-09-13T11:22:06.630 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T11:22:06.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-13T11:22:06.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:07.371 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-13T11:22:07.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T11:22:07.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:10.216 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T11:22:10.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-13T11:22:10.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:16.061 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-13T11:22:16.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-13T11:22:16.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:24.957 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-09-13T11:22:25.729 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-13T11:22:28.541 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-13T11:22:33.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-13T11:22:33.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:34.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-13T11:22:34.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T11:22:34.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:37.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-09-13T11:22:37.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-13T11:22:37.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:37.789 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-13T11:22:40.605 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-13T11:22:43.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-13T11:22:43.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:46.497 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-09-13T11:22:49.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-13T11:22:49.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:49.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T11:22:49.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:52.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-13T11:22:52.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:22:58.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-13T11:22:58.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:23:21.026 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-09-13T11:23:21.394 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-13T11:23:21.394 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:23:21.394 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-09-13T11:23:21.410 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:23:21.410 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:23:21.410 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:23:21.410 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:23:22.411 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T11:23:22.411 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:23:22.516 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-09-13T11:23:22.537 DEBUG:teuthology.parallel:result is None 2024-09-13T11:23:49.793 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T11:23:52.610 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T11:23:58.503 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T11:24:04.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T11:24:04.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:24:05.270 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-13T11:24:05.619 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T11:24:05.619 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:24:05.619 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-13T11:24:05.635 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:24:05.636 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:24:05.636 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:24:05.636 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:24:06.637 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-13T11:24:06.637 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:24:06.720 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-09-13T11:24:06.743 DEBUG:teuthology.parallel:result is None 2024-09-13T11:24:07.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-13T11:24:07.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:24:07.947 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-09-13T11:24:08.323 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-13T11:24:08.323 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:24:08.323 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-09-13T11:24:08.341 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:24:08.341 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:24:08.341 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:24:08.341 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:24:09.342 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T11:24:09.342 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:24:09.430 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-09-13T11:24:09.453 DEBUG:teuthology.parallel:result is None 2024-09-13T11:24:13.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-13T11:24:13.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:24:13.912 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-13T11:24:14.249 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-13T11:24:14.249 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-13T11:24:14.249 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-09-13T11:24:14.265 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-505.el9.x86_64 2024-09-13T11:24:14.266 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T11:24:14.266 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T11:24:14.266 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T11:24:15.267 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-13T11:24:15.267 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T11:24:15.374 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-09-13T11:24:15.390 DEBUG:teuthology.parallel:result is None 2024-09-13T11:24:15.390 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T11:24:15.399 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T11:24:15.399 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:24:15.402 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:24:15.405 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:24:15.407 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T11:24:15.426 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T11:24:15.452 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T11:24:15.459 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T11:24:15.460 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:24:15.464 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:24:15.466 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:24:15.469 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T11:24:15.544 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T11:24:15.550 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T11:24:15.550 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:24:15.556 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:24:15.559 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:24:15.561 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T11:24:15.592 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.592 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.595 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.603 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.603 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.604 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.607 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.614 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T11:24:15.616 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T11:24:15.623 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T11:24:15.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:24:15.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:24:15.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:24:15.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T11:24:15.685 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T11:24:15.700 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T11:24:15.701 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:24:15.718 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:24:15.721 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:24:15.724 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T11:24:15.742 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:15.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:15.864 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:15.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:15.991 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:24:15.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:24:16.058 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:16.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:16.161 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:16.246 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:16.288 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:24:16.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:24:16.354 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:16.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:16.461 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:16.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:16.548 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:24:16.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:24:16.616 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:16.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T11:24:16.716 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:16.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T11:24:16.844 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:24:16.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T11:24:16.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-09-13T11:24:16.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-09-13T11:24:16.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-09-13T11:24:16.919 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-09-13T11:24:16.957 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:24:16.969 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:24:16.971 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:24:16.996 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:24:17.468 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T11:24:17.475 INFO:teuthology.task.internal:Starting timer... 2024-09-13T11:24:17.476 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T11:24:17.486 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T11:24:17.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-09-13T11:24:17.590 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-09-13T11:24:17.592 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-09-13T11:24:17.671 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-09-13T11:24:17.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-09-13T11:24:17.736 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-09-13T11:24:17.738 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-09-13T11:24:17.804 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-09-13T11:24:17.806 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T11:24:17.806 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-09-13T11:24:17.831 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-09-13T11:24:17.831 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-09-13T11:24:17.858 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-09-13T11:24:17.858 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-09-13T11:24:17.887 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-09-13T11:24:17.887 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-09-13T11:24:17.922 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-09-13T11:24:17.922 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-09-13T11:24:17.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:24:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:24:17.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:24:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:24:17.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:24:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:24:18.016 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:24:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:24:18.046 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T11:24:18.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:24:18.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:24:18.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:24:18.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-09-13T11:24:18.181 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T11:24:18.191 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T11:24:18.194 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T11:24:18.306 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T11:24:18.316 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T11:24:18.317 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi118.front.sepia.ceph.com,smithi154.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T11:28:07.513 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-09-13T11:28:07.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-13T11:28:07.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:07.594 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-09-13T11:28:07.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-09-13T11:28:07.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-13T11:28:07.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:07.744 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-09-13T11:28:07.824 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-13T11:28:07.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T11:28:07.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:07.902 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-13T11:28:07.982 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-13T11:28:07.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-13T11:28:07.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T11:28:08.056 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-09-13T11:28:08.137 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-09-13T11:28:08.138 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T11:28:08.149 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T11:28:08.149 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:28:08.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:28:08.152 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:28:08.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:28:08.155 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:28:08.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:28:08.158 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T11:28:08.158 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:28:08.183 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:28:08.186 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:28:08.189 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:28:08.199 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:28:08.201 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:28:08.205 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:28:08.207 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T11:28:08.222 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T11:28:08.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-09-13T11:28:08.228 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-09-13T11:28:08.232 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-09-13T11:28:08.239 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-09-13T11:28:08.239 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-09-13T11:28:08.244 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-09-13T11:28:08.248 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-09-13T11:28:08.254 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:28:08.255 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:28:08.259 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-09-13T11:28:08.259 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:28:08.269 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:28:08.269 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:28:08.273 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T11:28:08.274 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:28:08.287 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T11:28:08.326 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:28:08.329 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:28:08.329 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-09-13T11:28:08.329 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.329 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.329 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.329 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.333 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:28:08.334 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:28:08.334 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-09-13T11:28:08.334 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.334 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.334 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.334 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.335 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:28:08.338 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:28:08.338 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-13T11:28:08.338 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.338 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.338 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.338 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.341 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:28:08.346 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:28:08.346 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-13T11:28:08.346 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.346 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.346 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.346 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:28:08.347 INFO:teuthology.run_tasks:Running task install... 2024-09-13T11:28:08.359 DEBUG:teuthology.task.install:project ceph 2024-09-13T11:28:08.359 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:28:08.359 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T11:28:08.359 INFO:teuthology.task.install:Using flavor: default 2024-09-13T11:28:08.369 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T11:28:08.370 INFO:teuthology.task.install:extra packages: [] 2024-09-13T11:28:08.371 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:28:08.371 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:28:08.374 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:28:08.374 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:28:08.375 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:28:08.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:28:08.376 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T11:28:08.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:28:08.537 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:28:08.537 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:28:08.541 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:28:08.541 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:28:08.548 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:28:08.548 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:28:08.551 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T11:28:08.551 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T11:28:08.649 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:28:08.650 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:28:08.650 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:28:08.660 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:28:08.661 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:28:08.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:28:08.663 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:28:08.663 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:28:08.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:28:08.668 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T11:28:08.668 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:28:08.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T11:28:08.676 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:28:08.676 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:28:08.689 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:28:08.689 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:28:08.690 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:28:08.691 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:28:08.694 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T11:28:08.694 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T11:28:08.741 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:28:08.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:28:08.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:28:08.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T11:28:08.816 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:28:08.833 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:28:08.835 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:28:08.835 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T11:28:08.881 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-09-13T11:28:08.882 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-09-13T11:28:08.898 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-09-13T11:28:08.899 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-09-13T11:28:08.899 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-09-13T11:28:08.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-09-13T11:28:08.901 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-09-13T11:28:08.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-09-13T11:28:09.112 INFO:teuthology.orchestra.run.smithi204.stdout:45 files removed 2024-09-13T11:28:09.129 INFO:teuthology.orchestra.run.smithi154.stdout:45 files removed 2024-09-13T11:28:09.130 INFO:teuthology.orchestra.run.smithi118.stdout:45 files removed 2024-09-13T11:28:09.138 INFO:teuthology.orchestra.run.smithi038.stdout:45 files removed 2024-09-13T11:28:09.139 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2024-09-13T11:28:09.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2024-09-13T11:28:09.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2024-09-13T11:28:09.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-09-13T11:28:09.710 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 297 kB/s | 84 kB 00:00 2024-09-13T11:28:09.711 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 313 kB/s | 84 kB 00:00 2024-09-13T11:28:09.716 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 309 kB/s | 84 kB 00:00 2024-09-13T11:28:09.746 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 299 kB/s | 84 kB 00:00 2024-09-13T11:28:10.003 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-13T11:28:10.004 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-13T11:28:10.008 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-13T11:28:10.046 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-13T11:28:10.278 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-13T11:28:10.283 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-13T11:28:10.286 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-13T11:28:10.330 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-13T11:28:10.753 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-13T11:28:10.859 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-13T11:28:11.128 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-13T11:28:12.156 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-13T11:28:12.730 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-13T11:28:12.835 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-13T11:28:16.021 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 20 MB 00:03 2024-09-13T11:28:17.164 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-13T11:28:17.973 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-13T11:28:18.819 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.5 MB 00:02 2024-09-13T11:28:18.923 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-13T11:28:20.053 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-13T11:28:20.361 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 88 kB/s | 19 kB 00:00 2024-09-13T11:28:20.430 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-13T11:28:21.149 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-13T11:28:22.106 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 19 kB 00:00 2024-09-13T11:28:22.424 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-13T11:28:22.977 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-13T11:28:23.867 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 120 kB/s | 19 kB 00:00 2024-09-13T11:28:24.929 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-13T11:28:25.467 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-09-13T11:28:26.596 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-13T11:28:27.825 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:28:27.834 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:28:27.834 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:28:27.834 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:28:27.834 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:28:27.834 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:28:27.835 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:28:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:28:27.837 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:28:27.837 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:28:27.837 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:28:27.837 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:28:27.837 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:28:27.837 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:28:27.837 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout:Install 30 Packages 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:27.838 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 62 M 2024-09-13T11:28:27.839 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:28:28.460 INFO:teuthology.orchestra.run.smithi118.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 9.6 MB/s | 5.5 MB 00:00 2024-09-13T11:28:28.474 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T11:28:28.518 INFO:teuthology.orchestra.run.smithi118.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 433 kB/s | 25 kB 00:00 2024-09-13T11:28:28.644 INFO:teuthology.orchestra.run.smithi118.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 5.9 MB/s | 755 kB 00:00 2024-09-13T11:28:28.727 INFO:teuthology.orchestra.run.smithi118.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 6.0 MB/s | 508 kB 00:00 2024-09-13T11:28:28.948 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:28:28.955 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:28:28.956 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:28:28.957 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout:Install 30 Packages 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:28.958 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 62 M 2024-09-13T11:28:28.959 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:28:28.986 INFO:teuthology.orchestra.run.smithi118.stdout:(5/32): ceph-common-19.3.0-4697.gd58f7306.el9.x 20 MB/s | 22 MB 00:01 2024-09-13T11:28:29.103 INFO:teuthology.orchestra.run.smithi118.stdout:(6/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 9.0 MB/s | 11 MB 00:01 2024-09-13T11:28:29.128 INFO:teuthology.orchestra.run.smithi118.stdout:(7/32): python3-ceph-argparse-19.3.0-4697.gd58f 318 kB/s | 45 kB 00:00 2024-09-13T11:28:29.164 INFO:teuthology.orchestra.run.smithi118.stdout:(8/32): python3-ceph-common-19.3.0-4697.gd58f73 2.4 MB/s | 143 kB 00:00 2024-09-13T11:28:29.187 INFO:teuthology.orchestra.run.smithi118.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 2.7 MB/s | 161 kB 00:00 2024-09-13T11:28:29.213 INFO:teuthology.orchestra.run.smithi118.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 6.3 MB/s | 321 kB 00:00 2024-09-13T11:28:29.238 INFO:teuthology.orchestra.run.smithi118.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.9 MB/s | 305 kB 00:00 2024-09-13T11:28:29.263 INFO:teuthology.orchestra.run.smithi118.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T11:28:29.355 INFO:teuthology.orchestra.run.smithi118.stdout:(13/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_ 8.8 MB/s | 5.5 MB 00:00 2024-09-13T11:28:29.414 INFO:teuthology.orchestra.run.smithi118.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 608 kB/s | 107 kB 00:00 2024-09-13T11:28:29.464 INFO:teuthology.orchestra.run.smithi118.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2024-09-13T11:28:29.490 INFO:teuthology.orchestra.run.smithi118.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 708 kB/s | 160 kB 00:00 2024-09-13T11:28:29.548 INFO:teuthology.orchestra.run.smithi118.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-13T11:28:29.574 INFO:teuthology.orchestra.run.smithi118.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 506 kB/s | 42 kB 00:00 2024-09-13T11:28:29.599 INFO:teuthology.orchestra.run.smithi118.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-13T11:28:29.624 INFO:teuthology.orchestra.run.smithi118.stdout:(20/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-09-13T11:28:29.649 INFO:teuthology.orchestra.run.smithi118.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-13T11:28:29.666 INFO:teuthology.orchestra.run.smithi118.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 706 kB/s | 47 kB 00:00 2024-09-13T11:28:29.725 INFO:teuthology.orchestra.run.smithi118.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-13T11:28:29.742 INFO:teuthology.orchestra.run.smithi118.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-09-13T11:28:29.767 INFO:teuthology.orchestra.run.smithi118.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-09-13T11:28:29.784 INFO:teuthology.orchestra.run.smithi118.stdout:(26/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 252 kB/s | 26 kB 00:00 2024-09-13T11:28:29.806 INFO:teuthology.orchestra.run.smithi154.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 12 MB/s | 5.5 MB 00:00 2024-09-13T11:28:29.834 INFO:teuthology.orchestra.run.smithi118.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-13T11:28:29.860 INFO:teuthology.orchestra.run.smithi118.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-13T11:28:29.864 INFO:teuthology.orchestra.run.smithi154.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 433 kB/s | 25 kB 00:00 2024-09-13T11:28:29.927 INFO:teuthology.orchestra.run.smithi118.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-13T11:28:29.956 INFO:teuthology.orchestra.run.smithi154.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-13T11:28:30.002 INFO:teuthology.orchestra.run.smithi118.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-13T11:28:30.057 INFO:teuthology.orchestra.run.smithi154.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 5.0 MB/s | 508 kB 00:00 2024-09-13T11:28:30.278 INFO:teuthology.orchestra.run.smithi118.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 8.3 MB/s | 3.5 MB 00:00 2024-09-13T11:28:30.553 INFO:teuthology.orchestra.run.smithi118.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 5.2 MB/s | 3.2 MB 00:00 2024-09-13T11:28:30.557 INFO:teuthology.orchestra.run.smithi154.stdout:(5/32): ceph-common-19.3.0-4697.gd58f7306.el9.x 18 MB/s | 22 MB 00:01 2024-09-13T11:28:30.558 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:28:30.558 INFO:teuthology.orchestra.run.smithi118.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-13T11:28:30.663 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:28:30.682 INFO:teuthology.orchestra.run.smithi154.stdout:(6/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 8.2 MB/s | 11 MB 00:01 2024-09-13T11:28:30.698 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:28:30.698 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:28:30.702 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T11:28:30.707 INFO:teuthology.orchestra.run.smithi154.stdout:(7/32): python3-ceph-argparse-19.3.0-4697.gd58f 301 kB/s | 45 kB 00:00 2024-09-13T11:28:30.733 INFO:teuthology.orchestra.run.smithi154.stdout:(8/32): python3-ceph-common-19.3.0-4697.gd58f73 2.8 MB/s | 143 kB 00:00 2024-09-13T11:28:30.758 INFO:teuthology.orchestra.run.smithi154.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 3.2 MB/s | 161 kB 00:00 2024-09-13T11:28:30.783 INFO:teuthology.orchestra.run.smithi154.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 6.2 MB/s | 321 kB 00:00 2024-09-13T11:28:30.817 INFO:teuthology.orchestra.run.smithi154.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.1 MB/s | 305 kB 00:00 2024-09-13T11:28:30.834 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:28:30.841 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:28:30.841 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:28:30.841 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:28:30.841 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:28:30.841 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:28:30.841 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-09-13T11:28:30.841 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:28:30.842 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:28:30.843 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout:Install 30 Packages 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2024-09-13T11:28:30.844 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:30.845 INFO:teuthology.orchestra.run.smithi154.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.7 MB/s | 99 kB 00:00 2024-09-13T11:28:30.845 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 62 M 2024-09-13T11:28:30.845 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:28:30.917 INFO:teuthology.orchestra.run.smithi154.stdout:(13/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_ 6.4 MB/s | 5.5 MB 00:00 2024-09-13T11:28:31.059 INFO:teuthology.orchestra.run.smithi154.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 441 kB/s | 107 kB 00:00 2024-09-13T11:28:31.076 INFO:teuthology.orchestra.run.smithi154.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 285 kB/s | 45 kB 00:00 2024-09-13T11:28:31.102 INFO:teuthology.orchestra.run.smithi154.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 618 kB/s | 160 kB 00:00 2024-09-13T11:28:31.152 INFO:teuthology.orchestra.run.smithi154.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-13T11:28:31.186 INFO:teuthology.orchestra.run.smithi154.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-13T11:28:31.244 INFO:teuthology.orchestra.run.smithi154.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-13T11:28:31.269 INFO:teuthology.orchestra.run.smithi154.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 429 kB/s | 36 kB 00:00 2024-09-13T11:28:31.295 INFO:teuthology.orchestra.run.smithi154.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-09-13T11:28:31.328 INFO:teuthology.orchestra.run.smithi154.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 566 kB/s | 47 kB 00:00 2024-09-13T11:28:31.379 INFO:teuthology.orchestra.run.smithi154.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T11:28:31.387 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:28:31.387 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:28:31.412 INFO:teuthology.orchestra.run.smithi154.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 353 kB/s | 26 kB 00:00 2024-09-13T11:28:31.437 INFO:teuthology.orchestra.run.smithi154.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 857 kB/s | 49 kB 00:00 2024-09-13T11:28:31.463 INFO:teuthology.orchestra.run.smithi154.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T11:28:31.505 INFO:teuthology.orchestra.run.smithi154.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-13T11:28:31.622 INFO:teuthology.orchestra.run.smithi154.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-09-13T11:28:31.697 INFO:teuthology.orchestra.run.smithi154.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-13T11:28:31.739 INFO:teuthology.orchestra.run.smithi154.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-09-13T11:28:31.881 INFO:teuthology.orchestra.run.smithi154.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-13T11:28:32.023 INFO:teuthology.orchestra.run.smithi154.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 9.9 MB/s | 3.2 MB 00:00 2024-09-13T11:28:32.029 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:28:32.029 INFO:teuthology.orchestra.run.smithi154.stdout:Total 20 MB/s | 62 MB 00:03 2024-09-13T11:28:32.137 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:28:32.173 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:28:32.173 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:28:32.226 INFO:teuthology.orchestra.run.smithi204.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 5.7 MB/s | 5.5 MB 00:00 2024-09-13T11:28:32.267 INFO:teuthology.orchestra.run.smithi204.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 607 kB/s | 25 kB 00:00 2024-09-13T11:28:32.393 INFO:teuthology.orchestra.run.smithi204.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 5.9 MB/s | 755 kB 00:00 2024-09-13T11:28:32.568 INFO:teuthology.orchestra.run.smithi204.stdout:(4/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 8.3 MB/s | 11 MB 00:01 2024-09-13T11:28:32.593 INFO:teuthology.orchestra.run.smithi204.stdout:(5/32): libradosstriper1-19.3.0-4697.gd58f7306. 2.5 MB/s | 508 kB 00:00 2024-09-13T11:28:32.652 INFO:teuthology.orchestra.run.smithi204.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 775 kB/s | 45 kB 00:00 2024-09-13T11:28:32.711 INFO:teuthology.orchestra.run.smithi204.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 2.4 MB/s | 143 kB 00:00 2024-09-13T11:28:32.786 INFO:teuthology.orchestra.run.smithi204.stdout:(8/32): python3-cephfs-19.3.0-4697.gd58f7306.el 2.1 MB/s | 161 kB 00:00 2024-09-13T11:28:32.878 INFO:teuthology.orchestra.run.smithi204.stdout:(9/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 18 MB/s | 5.5 MB 00:00 2024-09-13T11:28:32.879 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:28:32.879 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:28:32.900 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:28:32.903 INFO:teuthology.orchestra.run.smithi204.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 2.7 MB/s | 321 kB 00:00 2024-09-13T11:28:32.937 INFO:teuthology.orchestra.run.smithi204.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.1 MB/s | 305 kB 00:00 2024-09-13T11:28:32.962 INFO:teuthology.orchestra.run.smithi204.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.7 MB/s | 99 kB 00:00 2024-09-13T11:28:32.992 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:28:33.033 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:28:33.108 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:28:33.131 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:33.139 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:28:33.151 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:28:33.155 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T11:28:33.156 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T11:28:33.157 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout:Install 30 Packages 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-09-13T11:28:33.158 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:28:33.159 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 62 M 2024-09-13T11:28:33.159 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:28:33.187 INFO:teuthology.orchestra.run.smithi204.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 11 MB/s | 22 MB 00:01 2024-09-13T11:28:33.246 INFO:teuthology.orchestra.run.smithi204.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-09-13T11:28:33.271 INFO:teuthology.orchestra.run.smithi204.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 542 kB/s | 45 kB 00:00 2024-09-13T11:28:33.322 INFO:teuthology.orchestra.run.smithi204.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 445 kB/s | 160 kB 00:00 2024-09-13T11:28:33.342 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:33.347 INFO:teuthology.orchestra.run.smithi204.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-13T11:28:33.373 INFO:teuthology.orchestra.run.smithi204.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 842 kB/s | 42 kB 00:00 2024-09-13T11:28:33.404 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:33.406 INFO:teuthology.orchestra.run.smithi204.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-13T11:28:33.431 INFO:teuthology.orchestra.run.smithi204.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-13T11:28:33.457 INFO:teuthology.orchestra.run.smithi204.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-09-13T11:28:33.482 INFO:teuthology.orchestra.run.smithi204.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-13T11:28:33.541 INFO:teuthology.orchestra.run.smithi204.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T11:28:33.566 INFO:teuthology.orchestra.run.smithi204.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-13T11:28:33.591 INFO:teuthology.orchestra.run.smithi204.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 277 kB/s | 26 kB 00:00 2024-09-13T11:28:33.616 INFO:teuthology.orchestra.run.smithi204.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T11:28:33.642 INFO:teuthology.orchestra.run.smithi204.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-13T11:28:33.670 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-13T11:28:33.683 INFO:teuthology.orchestra.run.smithi204.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-13T11:28:33.729 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 431 kB/s | 25 kB 00:00 2024-09-13T11:28:33.733 INFO:teuthology.orchestra.run.smithi204.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-13T11:28:33.807 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:28:33.821 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-13T11:28:33.834 INFO:teuthology.orchestra.run.smithi204.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-13T11:28:33.860 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:33.904 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): libradosstriper1-19.3.0-4697.gd58f7306. 6.0 MB/s | 508 kB 00:00 2024-09-13T11:28:33.945 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:34.034 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:28:34.081 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:34.125 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:34.153 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:28:34.197 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:28:34.217 INFO:teuthology.orchestra.run.smithi204.stdout:(31/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 6.7 MB/s | 3.2 MB 00:00 2024-09-13T11:28:34.268 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:34.279 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 11 MB/s | 11 MB 00:01 2024-09-13T11:28:34.298 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:34.357 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:28:34.362 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:34.363 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 12 MB/s | 5.5 MB 00:00 2024-09-13T11:28:34.380 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): python3-ceph-argparse-19.3.0-4697.gd58f 451 kB/s | 45 kB 00:00 2024-09-13T11:28:34.397 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:28:34.413 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): python3-ceph-common-19.3.0-4697.gd58f73 2.8 MB/s | 143 kB 00:00 2024-09-13T11:28:34.431 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:28:34.434 INFO:teuthology.orchestra.run.smithi204.stdout:(32/32): librados2-19.3.0-4697.gd58f7306.el9.x8 4.6 MB/s | 3.5 MB 00:00 2024-09-13T11:28:34.440 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:28:34.440 INFO:teuthology.orchestra.run.smithi204.stdout:Total 17 MB/s | 62 MB 00:03 2024-09-13T11:28:34.447 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 2.3 MB/s | 161 kB 00:00 2024-09-13T11:28:34.461 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:28:34.472 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 5.4 MB/s | 321 kB 00:00 2024-09-13T11:28:34.488 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:28:34.498 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.9 MB/s | 305 kB 00:00 2024-09-13T11:28:34.509 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:34.520 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:28:34.523 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T11:28:34.531 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:34.543 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:28:34.547 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:28:34.579 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:28:34.579 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:28:34.589 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:28:34.629 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:28:34.658 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:28:34.678 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:34.697 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:28:34.730 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:34.740 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 14 MB/s | 22 MB 00:01 2024-09-13T11:28:35.014 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:28:35.062 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:35.107 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 123 kB/s | 45 kB 00:00 2024-09-13T11:28:35.132 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 169 kB/s | 107 kB 00:00 2024-09-13T11:28:35.135 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:28:35.153 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:35.199 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 237 kB/s | 160 kB 00:00 2024-09-13T11:28:35.202 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:28:35.204 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:35.246 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:28:35.270 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:28:35.271 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:28:35.306 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:35.307 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-09-13T11:28:35.399 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:28:35.422 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:35.424 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-09-13T11:28:35.486 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:35.523 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:28:35.533 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 331 kB/s | 36 kB 00:00 2024-09-13T11:28:35.567 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.4 MB/s | 662 kB 00:00 2024-09-13T11:28:35.574 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:28:35.617 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-13T11:28:35.642 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 660 kB/s | 221 kB 00:00 2024-09-13T11:28:35.653 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:35.668 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 353 kB/s | 47 kB 00:00 2024-09-13T11:28:35.675 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:35.709 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-09-13T11:28:35.731 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:35.735 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 825 kB/s | 49 kB 00:00 2024-09-13T11:28:35.735 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:28:35.760 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2024-09-13T11:28:35.765 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:28:35.774 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:35.802 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-13T11:28:35.844 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-13T11:28:35.845 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:35.875 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:28:35.911 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-13T11:28:35.916 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:28:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:28:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:28:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:28:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:28:36.003 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-13T11:28:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:28:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:28:36.153 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-13T11:28:36.212 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-13T11:28:36.217 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:28:36.217 INFO:teuthology.orchestra.run.smithi038.stdout:Total 20 MB/s | 62 MB 00:03 2024-09-13T11:28:36.316 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:28:36.356 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:28:36.356 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:28:36.379 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:28:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:36.544 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:28:36.579 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:28:36.760 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:28:36.844 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:28:36.873 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:28:36.898 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:28:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:37.071 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:28:37.074 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:28:37.074 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:28:37.109 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:37.148 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:37.163 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:37.203 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:37.235 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:28:37.269 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:28:37.606 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:28:37.666 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:37.760 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:37.850 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:28:37.876 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:37.932 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:37.968 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:28:38.013 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:28:38.083 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:38.120 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:38.177 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:38.219 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:28:38.281 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:38.331 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:38.340 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:28:38.371 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:28:38.412 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:28:38.452 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:28:38.481 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:28:38.523 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:28:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:28:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T11:28:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T11:28:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T11:28:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T11:28:38.847 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:28:38.894 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:38.968 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:28:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T11:28:39.042 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:28:39.094 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:39.094 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:28:39.094 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:28:39.095 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:39.128 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:39.443 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T11:28:39.502 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:39.547 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:28:39.578 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:39.579 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:39.659 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:39.677 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T11:28:39.703 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:39.738 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:28:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T11:28:39.782 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:28:39.793 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T11:28:39.840 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T11:28:39.934 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:39.964 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:40.028 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T11:28:40.063 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T11:28:40.173 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T11:28:40.206 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T11:28:40.246 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T11:28:40.286 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T11:28:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T11:28:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T11:28:40.409 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:40.443 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:40.538 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:40.538 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:28:40.539 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:28:40.539 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:40.570 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:40.696 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T11:28:40.753 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:40.828 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T11:28:40.858 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T11:28:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T11:28:41.413 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T11:28:41.439 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:41.510 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T11:28:41.582 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T11:28:41.624 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T11:28:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:42.145 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:42.744 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:42.782 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:42.872 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:42.872 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:28:42.872 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:28:42.872 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:42.903 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:44.114 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:44.154 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:44.771 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T11:28:44.815 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:44.908 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T11:28:44.909 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T11:28:44.909 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T11:28:44.909 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:28:44.950 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:50.209 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:50.209 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-09-13T11:28:50.209 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-09-13T11:28:50.209 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-09-13T11:28:50.210 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-09-13T11:28:50.210 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-09-13T11:28:50.210 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-09-13T11:28:50.210 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-09-13T11:28:50.210 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:50.832 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:50.832 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-09-13T11:28:50.833 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-09-13T11:28:50.833 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-09-13T11:28:50.833 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-09-13T11:28:50.833 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-09-13T11:28:50.833 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-09-13T11:28:50.833 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-09-13T11:28:50.833 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:51.234 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:51.266 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:51.266 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:28:51.266 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:28:51.267 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:51.267 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:51.267 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:51.268 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:51.346 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:51.346 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:51.853 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:51.878 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:51.879 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:28:51.879 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:28:51.879 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:51.879 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:51.879 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:51.881 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:51.963 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:51.964 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:52.584 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:52.584 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:28:52.585 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:28:52.586 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:28:52.587 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:28:53.038 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:53.038 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-09-13T11:28:53.038 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-09-13T11:28:53.038 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-09-13T11:28:53.038 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-09-13T11:28:53.038 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-09-13T11:28:53.038 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-09-13T11:28:53.039 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-09-13T11:28:53.039 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:53.109 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:53.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:28:53.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:28:53.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:28:53.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:28:53.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:28:53.109 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:28:53.110 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:28:53.111 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:28:53.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:28:53.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:28:53.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:28:53.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:28:53.112 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:28:53.113 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:28:53.113 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:28:53.114 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:28:53.114 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:53.114 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:28:53.114 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:28:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:53.720 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-09-13T11:28:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.720 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:28:53.721 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:53.722 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:28:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:28:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:28:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:28:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:28:53.723 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:53.723 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:28:54.060 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:54.060 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:54.060 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-09-13T11:28:54.061 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.061 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.061 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:28:54.061 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:28:54.061 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.061 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.061 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.062 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:28:54.063 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:28:54.064 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:28:54.064 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:28:54.064 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:54.064 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:28:54.070 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:54.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2024-09-13T11:28:54.096 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:54.096 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:28:54.096 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:28:54.096 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:54.096 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:54.097 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:54.098 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:54.190 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:54.190 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:54.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2024-09-13T11:28:54.636 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:28:54.768 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:28:54.769 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:28:54.770 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:28:54.770 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:54.770 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:28:54.770 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:28:54.770 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2024-09-13T11:28:54.770 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:28:54.770 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 51 M 2024-09-13T11:28:54.771 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 216 M 2024-09-13T11:28:54.771 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:28:55.041 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:28:55.111 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 723 kB/s | 161 kB 00:00 2024-09-13T11:28:55.175 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:28:55.175 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:55.176 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:28:55.177 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:28:55.177 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-09-13T11:28:55.177 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:28:55.177 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 51 M 2024-09-13T11:28:55.178 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 216 M 2024-09-13T11:28:55.178 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:28:55.328 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 691 kB/s | 304 kB 00:00 2024-09-13T11:28:55.361 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 255 kB/s | 64 kB 00:00 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:55.448 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:28:55.449 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:28:55.450 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:28:55.450 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:28:55.450 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:28:55.450 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:28:55.450 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:28:55.450 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:28:55.451 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:28:55.451 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:28:55.451 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:55.451 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:28:55.451 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:28:55.552 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 663 kB/s | 161 kB 00:00 2024-09-13T11:28:55.672 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:55.672 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-09-13T11:28:55.672 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-09-13T11:28:55.673 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-09-13T11:28:55.673 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-09-13T11:28:55.673 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-09-13T11:28:55.673 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-09-13T11:28:55.673 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-09-13T11:28:55.673 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:28:55.927 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-13T11:28:55.952 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 472 kB/s | 304 kB 00:00 2024-09-13T11:28:56.455 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:56.455 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:28:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:28:56.457 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:56.458 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:28:56.553 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 30 MB/s | 50 MB 00:01 2024-09-13T11:28:56.555 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:28:56.555 INFO:teuthology.orchestra.run.smithi118.stdout:Total 28 MB/s | 51 MB 00:01 2024-09-13T11:28:56.582 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:28:56.602 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:28:56.603 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:28:56.725 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:56.752 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T11:28:56.752 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:28:56.752 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T11:28:56.752 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:56.752 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T11:28:56.752 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:28:56.754 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:56.817 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2024-09-13T11:28:56.819 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T11:28:56.819 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:56.981 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:28:56.981 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:28:57.386 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:28:57.516 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:28:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:28:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:28:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:28:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-09-13T11:28:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:28:57.525 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 51 M 2024-09-13T11:28:57.525 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 216 M 2024-09-13T11:28:57.525 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:28:57.615 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:28:57.662 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:28:57.688 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:28:57.803 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-13T11:28:57.827 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 20 MB/s | 50 MB 00:02 2024-09-13T11:28:57.829 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:28:57.830 INFO:teuthology.orchestra.run.smithi154.stdout:Total 19 MB/s | 51 MB 00:02 2024-09-13T11:28:57.845 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-13T11:28:57.858 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:28:57.878 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:28:57.878 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:28:57.886 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 719 kB/s | 161 kB 00:00 2024-09-13T11:28:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T11:28:58.052 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T11:28:58.053 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T11:28:58.054 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T11:28:58.055 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T11:28:58.259 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:28:58.259 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:28:58.795 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:28:58.887 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:28:58.944 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:28:58.971 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.039 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.040 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T11:28:59.041 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T11:28:59.042 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T11:28:59.042 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:28:59.042 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:28:59.228 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-13T11:28:59.230 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:28:59.231 INFO:teuthology.orchestra.run.smithi204.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-13T11:28:59.259 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:28:59.279 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:28:59.279 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:28:59.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-09-13T11:28:59.665 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:28:59.665 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:28:59.985 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:00.134 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:00.134 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-09-13T11:29:00.135 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:00.136 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 51 M 2024-09-13T11:29:00.136 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 216 M 2024-09-13T11:29:00.136 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:29:00.195 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:00.300 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:29:00.366 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:29:00.447 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:29:00.617 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 722 kB/s | 161 kB 00:00 2024-09-13T11:29:01.191 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 381 kB/s | 304 kB 00:00 2024-09-13T11:29:01.225 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 106 kB/s | 64 kB 00:00 2024-09-13T11:29:01.975 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-13T11:29:01.978 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:01.978 INFO:teuthology.orchestra.run.smithi038.stdout:Total 28 MB/s | 51 MB 00:01 2024-09-13T11:29:02.006 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:29:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:29:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:29:02.431 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:29:02.431 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:29:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:29:02.561 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:03.011 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:29:03.100 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T11:29:03.165 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T11:29:03.202 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:29:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:29:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:05.206 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:29:05.288 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:29:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:29:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:29:05.644 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:05.645 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:05.955 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2024-09-13T11:29:06.505 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:06.505 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:29:06.505 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:29:06.505 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:29:06.531 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:41 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:06.676 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:06.679 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:06.679 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:29:06.679 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:06.679 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:29:06.679 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:29:06.679 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:29:06.679 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:29:06.680 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:29:06.681 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:06.682 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:29:06.683 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:06.683 INFO:teuthology.orchestra.run.smithi118.stdout:Install 37 Packages 2024-09-13T11:29:06.683 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:06.683 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 31 M 2024-09-13T11:29:06.683 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 109 M 2024-09-13T11:29:06.684 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:29:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:29:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:06.785 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:29:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:29:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:29:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:06.786 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:07.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2024-09-13T11:29:07.122 INFO:teuthology.orchestra.run.smithi118.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 45 kB/s | 6.6 kB 00:00 2024-09-13T11:29:07.281 INFO:teuthology.orchestra.run.smithi118.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-13T11:29:07.331 INFO:teuthology.orchestra.run.smithi118.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.7 MB/s | 2.4 MB 00:00 2024-09-13T11:29:07.440 INFO:teuthology.orchestra.run.smithi118.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 2.6 MB/s | 291 kB 00:00 2024-09-13T11:29:07.532 INFO:teuthology.orchestra.run.smithi118.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-13T11:29:07.607 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:41 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:07.649 INFO:teuthology.orchestra.run.smithi118.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-13T11:29:07.691 INFO:teuthology.orchestra.run.smithi118.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-13T11:29:07.723 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:07.723 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:29:07.724 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:29:07.724 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:29:07.733 INFO:teuthology.orchestra.run.smithi118.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-13T11:29:07.745 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:07.747 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:07.747 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:29:07.747 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:29:07.748 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:29:07.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:29:07.750 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:07.751 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:29:07.751 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:07.751 INFO:teuthology.orchestra.run.smithi154.stdout:Install 37 Packages 2024-09-13T11:29:07.751 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:07.751 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 31 M 2024-09-13T11:29:07.751 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 109 M 2024-09-13T11:29:07.752 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:29:07.775 INFO:teuthology.orchestra.run.smithi118.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-13T11:29:07.834 INFO:teuthology.orchestra.run.smithi118.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-13T11:29:07.876 INFO:teuthology.orchestra.run.smithi118.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 3.6 MB/s | 1.2 MB 00:00 2024-09-13T11:29:08.070 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:29:08.070 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:08.070 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:08.070 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:08.070 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:08.070 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:29:08.070 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:29:08.071 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:08.071 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:08.076 INFO:teuthology.orchestra.run.smithi118.stdout:(12/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86 22 MB/s | 17 MB 00:00 2024-09-13T11:29:08.135 INFO:teuthology.orchestra.run.smithi118.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 627 kB/s | 188 kB 00:00 2024-09-13T11:29:08.160 INFO:teuthology.orchestra.run.smithi118.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 417 kB/s | 35 kB 00:00 2024-09-13T11:29:08.185 INFO:teuthology.orchestra.run.smithi118.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 556 kB/s | 172 kB 00:00 2024-09-13T11:29:08.211 INFO:teuthology.orchestra.run.smithi118.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-13T11:29:08.236 INFO:teuthology.orchestra.run.smithi118.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 673 kB/s | 51 kB 00:00 2024-09-13T11:29:08.245 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T11:29:08.261 INFO:teuthology.orchestra.run.smithi118.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-13T11:29:08.303 INFO:teuthology.orchestra.run.smithi118.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-13T11:29:08.320 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:08.328 INFO:teuthology.orchestra.run.smithi118.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 651 kB/s | 43 kB 00:00 2024-09-13T11:29:08.342 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2024-09-13T11:29:08.362 INFO:teuthology.orchestra.run.smithi118.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.0 MB/s | 172 kB 00:00 2024-09-13T11:29:08.387 INFO:teuthology.orchestra.run.smithi118.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 126 kB/s | 22 kB 00:00 2024-09-13T11:29:08.413 INFO:teuthology.orchestra.run.smithi118.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2024-09-13T11:29:08.447 INFO:teuthology.orchestra.run.smithi118.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 874 kB/s | 23 kB 00:00 2024-09-13T11:29:08.472 INFO:teuthology.orchestra.run.smithi118.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 841 kB/s | 19 kB 00:00 2024-09-13T11:29:08.478 INFO:teuthology.orchestra.run.smithi154.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 54 kB/s | 6.6 kB 00:00 2024-09-13T11:29:08.497 INFO:teuthology.orchestra.run.smithi118.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-09-13T11:29:08.522 INFO:teuthology.orchestra.run.smithi118.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-13T11:29:08.548 INFO:teuthology.orchestra.run.smithi118.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-13T11:29:08.581 INFO:teuthology.orchestra.run.smithi118.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-13T11:29:08.607 INFO:teuthology.orchestra.run.smithi118.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 937 kB/s | 79 kB 00:00 2024-09-13T11:29:08.640 INFO:teuthology.orchestra.run.smithi118.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-09-13T11:29:08.657 INFO:teuthology.orchestra.run.smithi118.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-13T11:29:08.683 INFO:teuthology.orchestra.run.smithi118.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-13T11:29:08.687 INFO:teuthology.orchestra.run.smithi154.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 4.6 MB/s | 1.5 MB 00:00 2024-09-13T11:29:08.708 INFO:teuthology.orchestra.run.smithi118.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T11:29:08.733 INFO:teuthology.orchestra.run.smithi118.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T11:29:08.737 INFO:teuthology.orchestra.run.smithi154.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.2 MB/s | 2.4 MB 00:00 2024-09-13T11:29:08.759 INFO:teuthology.orchestra.run.smithi118.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-13T11:29:08.784 INFO:teuthology.orchestra.run.smithi118.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-13T11:29:08.788 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:08.788 INFO:teuthology.orchestra.run.smithi118.stdout:Total 15 MB/s | 31 MB 00:02 2024-09-13T11:29:08.841 INFO:teuthology.orchestra.run.smithi154.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 2.7 MB/s | 291 kB 00:00 2024-09-13T11:29:08.909 INFO:teuthology.orchestra.run.smithi154.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-13T11:29:08.918 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:40 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:08.942 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:29:08.983 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:29:08.983 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:29:09.058 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:09.061 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:29:09.061 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:09.061 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:09.061 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:29:09.062 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:29:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:29:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:29:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:29:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:29:09.062 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:29:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:29:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:29:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:29:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:29:09.063 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:29:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:29:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:29:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:29:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:29:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:29:09.064 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections 2024-09-13T11:29:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:29:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:29:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:29:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:29:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:29:09.065 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:29:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:29:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:29:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:29:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:29:09.066 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:29:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:29:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:29:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:29:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:29:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:29:09.067 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout:Install 37 Packages 2024-09-13T11:29:09.068 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:09.069 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 31 M 2024-09-13T11:29:09.069 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 109 M 2024-09-13T11:29:09.069 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:09.283 INFO:teuthology.orchestra.run.smithi154.stdout:(6/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-13T11:29:09.487 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:29:09.488 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:29:09.894 INFO:teuthology.orchestra.run.smithi154.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 240 kB/s | 253 kB 00:01 2024-09-13T11:29:10.050 INFO:teuthology.orchestra.run.smithi154.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-09-13T11:29:10.075 INFO:teuthology.orchestra.run.smithi154.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 137 kB/s | 106 kB 00:00 2024-09-13T11:29:10.106 INFO:teuthology.orchestra.run.smithi204.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-13T11:29:10.315 INFO:teuthology.orchestra.run.smithi204.stdout:(2/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.9 MB/s | 2.4 MB 00:00 2024-09-13T11:29:10.357 INFO:teuthology.orchestra.run.smithi204.stdout:(3/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-09-13T11:29:10.465 INFO:teuthology.orchestra.run.smithi204.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 2.6 MB/s | 291 kB 00:00 2024-09-13T11:29:10.541 INFO:teuthology.orchestra.run.smithi204.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-13T11:29:10.659 INFO:teuthology.orchestra.run.smithi154.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 177 kB/s | 135 kB 00:00 2024-09-13T11:29:10.682 INFO:teuthology.orchestra.run.smithi204.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-13T11:29:10.724 INFO:teuthology.orchestra.run.smithi204.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-09-13T11:29:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T11:29:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T11:29:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T11:29:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T11:29:10.752 INFO:teuthology.orchestra.run.smithi204.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-09-13T11:29:10.759 INFO:teuthology.orchestra.run.smithi154.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 176 kB/s | 125 kB 00:00 2024-09-13T11:29:10.775 INFO:teuthology.orchestra.run.smithi204.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-13T11:29:10.809 INFO:teuthology.orchestra.run.smithi204.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-13T11:29:10.826 INFO:teuthology.orchestra.run.smithi154.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 286 kB/s | 215 kB 00:00 2024-09-13T11:29:10.834 INFO:teuthology.orchestra.run.smithi204.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-13T11:29:10.860 INFO:teuthology.orchestra.run.smithi154.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 941 kB/s | 188 kB 00:00 2024-09-13T11:29:10.910 INFO:teuthology.orchestra.run.smithi154.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-13T11:29:10.911 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:29:10.935 INFO:teuthology.orchestra.run.smithi154.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-13T11:29:10.961 INFO:teuthology.orchestra.run.smithi154.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-09-13T11:29:10.974 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:29:10.986 INFO:teuthology.orchestra.run.smithi154.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-13T11:29:11.003 INFO:teuthology.orchestra.run.smithi154.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 620 kB/s | 42 kB 00:00 2024-09-13T11:29:11.006 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:29:11.009 INFO:teuthology.orchestra.run.smithi204.stdout:(12/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86 25 MB/s | 17 MB 00:00 2024-09-13T11:29:11.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T11:29:11.014 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:11.014 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:29:11.014 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:11.015 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:11.015 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T11:29:11.015 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T11:29:11.015 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:11.015 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:11.043 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:29:11.044 INFO:teuthology.orchestra.run.smithi154.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-13T11:29:11.078 INFO:teuthology.orchestra.run.smithi154.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.3 MB/s | 172 kB 00:00 2024-09-13T11:29:11.095 INFO:teuthology.orchestra.run.smithi154.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-13T11:29:11.112 INFO:teuthology.orchestra.run.smithi154.stdout:(22/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 398 kB/s | 43 kB 00:00 2024-09-13T11:29:11.114 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:29:11.118 INFO:teuthology.orchestra.run.smithi204.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 322 kB/s | 35 kB 00:00 2024-09-13T11:29:11.129 INFO:teuthology.orchestra.run.smithi154.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 319 kB/s | 11 kB 00:00 2024-09-13T11:29:11.143 INFO:teuthology.orchestra.run.smithi204.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 564 kB/s | 188 kB 00:00 2024-09-13T11:29:11.150 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:29:11.163 INFO:teuthology.orchestra.run.smithi154.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-09-13T11:29:11.177 INFO:teuthology.orchestra.run.smithi204.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 503 kB/s | 172 kB 00:00 2024-09-13T11:29:11.183 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:29:11.188 INFO:teuthology.orchestra.run.smithi154.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 452 kB/s | 23 kB 00:00 2024-09-13T11:29:11.202 INFO:teuthology.orchestra.run.smithi204.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 871 kB/s | 51 kB 00:00 2024-09-13T11:29:11.213 INFO:teuthology.orchestra.run.smithi154.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-09-13T11:29:11.227 INFO:teuthology.orchestra.run.smithi204.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 707 kB/s | 77 kB 00:00 2024-09-13T11:29:11.238 INFO:teuthology.orchestra.run.smithi154.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 264 kB/s | 19 kB 00:00 2024-09-13T11:29:11.253 INFO:teuthology.orchestra.run.smithi204.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-09-13T11:29:11.264 INFO:teuthology.orchestra.run.smithi154.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-13T11:29:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:29:11.278 INFO:teuthology.orchestra.run.smithi204.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 413 kB/s | 42 kB 00:00 2024-09-13T11:29:11.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-09-13T11:29:11.289 INFO:teuthology.orchestra.run.smithi154.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-13T11:29:11.314 INFO:teuthology.orchestra.run.smithi154.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-13T11:29:11.328 INFO:teuthology.orchestra.run.smithi204.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-09-13T11:29:11.340 INFO:teuthology.orchestra.run.smithi154.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-13T11:29:11.349 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:29:11.353 INFO:teuthology.orchestra.run.smithi204.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-13T11:29:11.365 INFO:teuthology.orchestra.run.smithi154.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-13T11:29:11.379 INFO:teuthology.orchestra.run.smithi204.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-13T11:29:11.390 INFO:teuthology.orchestra.run.smithi154.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-13T11:29:11.394 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:29:11.413 INFO:teuthology.orchestra.run.smithi204.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 526 kB/s | 18 kB 00:00 2024-09-13T11:29:11.424 INFO:teuthology.orchestra.run.smithi154.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T11:29:11.437 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:29:11.438 INFO:teuthology.orchestra.run.smithi204.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-13T11:29:11.449 INFO:teuthology.orchestra.run.smithi154.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-13T11:29:11.463 INFO:teuthology.orchestra.run.smithi204.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-13T11:29:11.474 INFO:teuthology.orchestra.run.smithi154.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-13T11:29:11.476 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:29:11.489 INFO:teuthology.orchestra.run.smithi204.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-13T11:29:11.501 INFO:teuthology.orchestra.run.smithi154.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-13T11:29:11.504 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:11.505 INFO:teuthology.orchestra.run.smithi154.stdout:Total 8.2 MB/s | 31 MB 00:03 2024-09-13T11:29:11.510 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:29:11.514 INFO:teuthology.orchestra.run.smithi204.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-13T11:29:11.539 INFO:teuthology.orchestra.run.smithi204.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-13T11:29:11.544 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:29:11.565 INFO:teuthology.orchestra.run.smithi204.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-13T11:29:11.585 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:29:11.590 INFO:teuthology.orchestra.run.smithi204.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-13T11:29:11.607 INFO:teuthology.orchestra.run.smithi204.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 392 kB/s | 16 kB 00:00 2024-09-13T11:29:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:29:11.632 INFO:teuthology.orchestra.run.smithi204.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-13T11:29:11.649 INFO:teuthology.orchestra.run.smithi204.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 857 kB/s | 36 kB 00:00 2024-09-13T11:29:11.658 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:29:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:29:11.674 INFO:teuthology.orchestra.run.smithi204.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-13T11:29:11.697 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:29:11.697 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:29:11.708 INFO:teuthology.orchestra.run.smithi204.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T11:29:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:29:11.733 INFO:teuthology.orchestra.run.smithi204.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-09-13T11:29:11.749 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:29:11.758 INFO:teuthology.orchestra.run.smithi204.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-13T11:29:11.764 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:11.764 INFO:teuthology.orchestra.run.smithi204.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-13T11:29:11.825 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:29:11.870 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:41 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:12.011 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:29:12.011 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:29:12.011 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:29:12.011 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:29:12.020 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:12.022 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:12.022 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:29:12.022 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T11:29:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T11:29:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T11:29:12.025 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout:Install 37 Packages 2024-09-13T11:29:12.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:12.027 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 31 M 2024-09-13T11:29:12.027 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 109 M 2024-09-13T11:29:12.027 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:29:12.067 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:29:12.129 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:29:12.168 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:29:12.193 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:29:12.193 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:29:12.234 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:29:12.289 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:29:12.426 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:29:12.426 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:29:12.475 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:29:12.527 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:29:12.568 INFO:teuthology.orchestra.run.smithi038.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 49 kB/s | 6.6 kB 00:00 2024-09-13T11:29:12.605 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:29:12.694 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:29:12.735 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:29:12.752 INFO:teuthology.orchestra.run.smithi038.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 4.8 MB/s | 1.5 MB 00:00 2024-09-13T11:29:12.811 INFO:teuthology.orchestra.run.smithi038.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.3 MB/s | 2.4 MB 00:00 2024-09-13T11:29:12.871 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:29:12.894 INFO:teuthology.orchestra.run.smithi038.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 3.4 MB/s | 291 kB 00:00 2024-09-13T11:29:13.028 INFO:teuthology.orchestra.run.smithi038.stdout:(5/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 10 MB/s | 4.8 MB 00:00 2024-09-13T11:29:13.049 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:29:13.087 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:13.114 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:13.114 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:13.114 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:29:13.114 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:13.114 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:13.114 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:13.119 INFO:teuthology.orchestra.run.smithi038.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-13T11:29:13.170 INFO:teuthology.orchestra.run.smithi038.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-13T11:29:13.345 INFO:teuthology.orchestra.run.smithi038.stdout:(8/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-13T11:29:13.371 INFO:teuthology.orchestra.run.smithi038.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 674 kB/s | 135 kB 00:00 2024-09-13T11:29:13.429 INFO:teuthology.orchestra.run.smithi038.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-13T11:29:13.463 INFO:teuthology.orchestra.run.smithi038.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-13T11:29:13.546 INFO:teuthology.orchestra.run.smithi038.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 622 kB/s | 125 kB 00:00 2024-09-13T11:29:13.565 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:29:13.597 INFO:teuthology.orchestra.run.smithi038.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-13T11:29:13.636 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:29:13.647 INFO:teuthology.orchestra.run.smithi038.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-13T11:29:13.672 INFO:teuthology.orchestra.run.smithi038.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 824 kB/s | 172 kB 00:00 2024-09-13T11:29:13.677 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:29:13.698 INFO:teuthology.orchestra.run.smithi038.stdout:(16/37): python3-markupsafe-1.1.1-12.el9.x86_64 230 kB/s | 35 kB 00:00 2024-09-13T11:29:13.705 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:29:13.723 INFO:teuthology.orchestra.run.smithi038.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-13T11:29:13.748 INFO:teuthology.orchestra.run.smithi038.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-13T11:29:13.763 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:13.776 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:29:13.798 INFO:teuthology.orchestra.run.smithi038.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-13T11:29:13.811 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:29:13.820 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:29:13.832 INFO:teuthology.orchestra.run.smithi038.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-09-13T11:29:13.852 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:29:13.853 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:29:13.857 INFO:teuthology.orchestra.run.smithi038.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 399 kB/s | 43 kB 00:00 2024-09-13T11:29:13.880 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:29:13.882 INFO:teuthology.orchestra.run.smithi038.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-13T11:29:13.907 INFO:teuthology.orchestra.run.smithi038.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 715 kB/s | 18 kB 00:00 2024-09-13T11:29:13.945 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:29:13.954 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:29:13.959 INFO:teuthology.orchestra.run.smithi038.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 454 kB/s | 23 kB 00:00 2024-09-13T11:29:13.995 INFO:teuthology.orchestra.run.smithi038.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-09-13T11:29:14.012 INFO:teuthology.orchestra.run.smithi038.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 435 kB/s | 19 kB 00:00 2024-09-13T11:29:14.025 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:29:14.029 INFO:teuthology.orchestra.run.smithi038.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-09-13T11:29:14.046 INFO:teuthology.orchestra.run.smithi038.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-13T11:29:14.055 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:29:14.057 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:29:14.071 INFO:teuthology.orchestra.run.smithi038.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-13T11:29:14.088 INFO:teuthology.orchestra.run.smithi038.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 986 kB/s | 16 kB 00:00 2024-09-13T11:29:14.106 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:29:14.113 INFO:teuthology.orchestra.run.smithi038.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-13T11:29:14.139 INFO:teuthology.orchestra.run.smithi038.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-13T11:29:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:29:14.148 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:29:14.164 INFO:teuthology.orchestra.run.smithi038.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-13T11:29:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:29:14.189 INFO:teuthology.orchestra.run.smithi038.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 9.1 MB/s | 230 kB 00:00 2024-09-13T11:29:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:29:14.214 INFO:teuthology.orchestra.run.smithi038.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-13T11:29:14.236 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:29:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:29:14.239 INFO:teuthology.orchestra.run.smithi038.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-09-13T11:29:14.279 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:29:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:29:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:29:14.323 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:29:14.331 INFO:teuthology.orchestra.run.smithi038.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-09-13T11:29:14.336 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:14.337 INFO:teuthology.orchestra.run.smithi038.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-13T11:29:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:29:14.357 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:29:14.385 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:29:14.407 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:29:14.418 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:29:14.447 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:29:14.467 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:29:14.495 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:29:14.502 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:29:14.519 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:29:14.540 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:29:14.540 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:29:14.545 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:29:14.593 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:29:14.613 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:14.634 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:29:14.641 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:29:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:14.641 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:14.660 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:29:14.706 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:29:14.738 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:29:14.799 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:29:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:29:14.846 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:29:14.903 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:29:14.925 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:29:14.967 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:29:14.978 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:29:15.025 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:29:15.030 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:29:15.031 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:29:15.073 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:29:15.099 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:15.123 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:15.123 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:15.124 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:29:15.124 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:15.124 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:15.124 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:15.137 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:29:15.161 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:29:15.217 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:29:15.295 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:29:15.323 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:29:15.358 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:15.374 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:29:15.382 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:15.382 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:15.382 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:29:15.382 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:15.382 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:15.382 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:15.383 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:29:15.424 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:29:15.452 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:29:15.455 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:15.540 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:29:15.552 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:29:15.582 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:29:15.708 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:29:15.721 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:29:15.751 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:15.778 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:15.778 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:15.778 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:15.779 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:15.870 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:29:15.901 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:15.927 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:15.928 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:15.928 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:29:15.928 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:15.928 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:15.928 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:29:16.476 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T11:29:16.517 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T11:29:16.554 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T11:29:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T11:29:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T11:29:16.705 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T11:29:16.798 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T11:29:16.876 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T11:29:16.931 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T11:29:16.974 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T11:29:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T11:29:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T11:29:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T11:29:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T11:29:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T11:29:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T11:29:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T11:29:17.260 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:17.288 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:17.288 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:17.288 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:29:17.288 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:17.288 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:17.288 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:17.301 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T11:29:17.388 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T11:29:17.402 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:29:17.410 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:29:17.411 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:29:17.411 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:29:17.411 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:29:17.412 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:29:17.429 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:17.429 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:17.429 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:29:17.429 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:17.429 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:17.430 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T11:29:17.632 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T11:29:17.691 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T11:29:17.745 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:17.754 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T11:29:17.769 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:17.769 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:17.769 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:29:17.769 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:17.769 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:17.769 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T11:29:17.877 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:17.885 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T11:29:17.901 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:17.901 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:17.901 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:29:17.901 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:17.901 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:17.902 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:18.019 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:18.019 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:18.019 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:29:18.019 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:18.019 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:18.019 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:18.083 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T11:29:18.128 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T11:29:18.155 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:18.155 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:18.155 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:29:18.155 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:18.155 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:18.156 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:18.216 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T11:29:18.217 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:18.305 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T11:29:18.342 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T11:29:18.478 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:29:18.479 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:29:18.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:29:18.481 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:18.482 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T11:29:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T11:29:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T11:29:18.729 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:18.729 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T11:29:18.729 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:18.729 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T11:29:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:18.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2024-09-13T11:29:19.471 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:19.559 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:19.601 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:19.602 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:29:19.602 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:19.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2024-09-13T11:29:19.966 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:29:19.967 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:29:19.968 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:29:19.969 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:29:19.981 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:29:19.982 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:29:19.983 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:29:19.984 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:29:19.984 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:29:19.985 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:29:20.255 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:20.303 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:20.332 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T11:29:20.332 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:20.332 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T11:29:20.333 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:20.333 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T11:29:20.333 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:20.387 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:29:20.388 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:20.389 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:29:20.389 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:20.389 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 783 k 2024-09-13T11:29:20.389 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 789 k 2024-09-13T11:29:20.389 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:29:20.651 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T11:29:20.651 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:20.651 INFO:teuthology.orchestra.run.smithi118.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-13T11:29:20.652 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:29:20.657 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:29:20.657 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:29:20.663 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:29:20.663 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:29:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:29:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:20.846 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T11:29:20.846 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:20.846 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T11:29:20.846 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:20.847 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T11:29:20.847 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:21.017 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:29:21.017 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:21.017 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:21.017 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.017 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:29:21.018 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:29:21.019 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:29:21.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:29:21.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:29:21.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:29:21.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:29:21.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:29:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:29:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:29:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:29:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:29:21.021 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:29:21.022 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:29:21.022 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:21.022 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:21.079 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:21.105 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T11:29:21.105 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:21.106 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T11:29:21.106 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:21.106 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T11:29:21.106 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:21.116 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:29:21.116 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:21.116 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:29:21.117 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.117 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.117 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.117 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:21.118 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.118 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:21.118 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:29:21.118 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:29:21.118 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:29:21.119 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:29:21.119 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:29:21.119 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:29:21.119 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:29:21.119 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:29:21.119 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:29:21.120 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:29:21.120 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:29:21.120 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:29:21.120 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:29:21.120 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:29:21.120 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:29:21.121 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:29:21.121 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:29:21.121 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:29:21.121 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:29:21.122 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:29:21.122 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:29:21.122 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:29:21.122 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:29:21.122 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:29:21.122 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:29:21.122 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:29:21.123 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:29:21.123 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:29:21.123 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:29:21.123 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:29:21.123 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:29:21.123 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:21.123 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:21.155 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:21.171 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:21.246 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:21.421 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2024-09-13T11:29:21.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2024-09-13T11:29:21.608 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:21.886 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:21.886 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:21.886 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:29:21.886 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:21.886 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:21.886 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:21.994 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:22.086 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:22.101 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:22.128 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:22.129 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:29:22.129 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:22.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:29:22.192 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:22.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2024-09-13T11:29:22.235 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:22.235 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:29:22.235 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:22.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2024-09-13T11:29:22.723 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:22.760 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:22.857 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:22.858 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-09-13T11:29:22.858 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:22.858 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:29:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:29:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:22.859 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:29:22.859 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:22.859 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:29:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:22.859 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 145 k 2024-09-13T11:29:22.859 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 432 k 2024-09-13T11:29:22.859 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:29:22.875 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:22.894 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:22.894 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:22.894 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:29:22.895 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:22.896 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 783 k 2024-09-13T11:29:22.896 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 789 k 2024-09-13T11:29:22.896 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:23.007 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:29:23.008 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:23.009 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:29:23.009 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:23.010 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 783 k 2024-09-13T11:29:23.010 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 789 k 2024-09-13T11:29:23.010 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:29:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T11:29:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T11:29:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T11:29:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T11:29:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T11:29:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T11:29:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T11:29:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T11:29:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T11:29:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T11:29:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T11:29:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T11:29:23.066 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 701 kB/s | 145 kB 00:00 2024-09-13T11:29:23.067 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:23.067 INFO:teuthology.orchestra.run.smithi118.stdout:Total 695 kB/s | 145 kB 00:00 2024-09-13T11:29:23.068 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:29:23.075 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:29:23.075 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:29:23.132 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:29:23.132 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:29:23.163 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T11:29:23.164 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:23.164 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.8 MB/s | 783 kB 00:00 2024-09-13T11:29:23.164 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:29:23.169 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:29:23.169 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:29:23.175 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:29:23.175 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:29:23.247 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:23.270 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.9 MB/s | 783 kB 00:00 2024-09-13T11:29:23.271 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:23.271 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-13T11:29:23.271 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:29:23.276 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:29:23.277 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:29:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:29:23.282 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:29:23.283 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:29:23.353 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:29:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:23.410 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:23.410 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:29:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:23.774 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:23.784 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:23.859 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:23.882 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:23.960 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T11:29:23.961 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T11:29:23.962 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T11:29:23.963 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:23.964 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:24.205 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:24.253 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:24.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-09-13T11:29:24.457 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:24.458 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:29:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:24.459 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:24.509 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:24.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:24.509 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:29:24.509 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:24.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:24.509 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:24.663 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:29:24.711 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2024-09-13T11:29:24.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:29:24.980 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:25.073 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:25.118 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:25.119 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:29:25.120 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:25.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-09-13T11:29:25.238 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:25.288 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:25.317 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:25.371 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:25.371 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 145 k 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 432 k 2024-09-13T11:29:25.372 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:25.379 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:25.421 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:25.422 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:29:25.422 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:25.452 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:25.453 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T11:29:25.453 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:25.453 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:29:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 145 k 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 432 k 2024-09-13T11:29:25.454 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:29:25.504 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:29:25.586 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 679 kB/s | 145 kB 00:00 2024-09-13T11:29:25.587 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:25.587 INFO:teuthology.orchestra.run.smithi204.stdout:Total 675 kB/s | 145 kB 00:00 2024-09-13T11:29:25.587 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:29:25.594 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:29:25.595 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:29:25.644 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 771 kB/s | 145 kB 00:00 2024-09-13T11:29:25.644 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:25.644 INFO:teuthology.orchestra.run.smithi154.stdout:Total 763 kB/s | 145 kB 00:00 2024-09-13T11:29:25.644 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:29:25.651 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:29:25.652 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:29:25.652 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:29:25.653 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:29:25.708 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:29:25.708 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:29:25.784 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:55 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:25.799 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:29:25.896 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:25.921 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:25.921 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:29:25.921 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:25.922 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:25.922 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:25.922 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:29:25.922 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:25.923 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 783 k 2024-09-13T11:29:25.924 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 789 k 2024-09-13T11:29:25.924 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:29:25.936 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:25.961 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:25.961 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:25.961 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:29:25.961 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:26.066 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:26.199 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:26.200 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:29:26.200 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:26.200 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:29:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:29:26.200 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:29:26.200 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:29:26.201 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:29:26.202 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:26.202 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:29:26.202 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:26.202 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-09-13T11:29:26.202 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:26.204 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 19 M 2024-09-13T11:29:26.204 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 159 M 2024-09-13T11:29:26.204 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:29:26.204 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.7 MB/s | 783 kB 00:00 2024-09-13T11:29:26.204 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:26.204 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.7 MB/s | 783 kB 00:00 2024-09-13T11:29:26.204 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:29:26.209 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:29:26.209 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:29:26.216 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:29:26.216 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:29:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:29:26.588 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 191 kB/s | 34 kB 00:00 2024-09-13T11:29:26.613 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 81 kB/s | 16 kB 00:00 2024-09-13T11:29:26.775 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:26.815 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:26.815 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:26.815 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:26.815 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:26.815 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:26.815 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:26.830 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-13T11:29:26.866 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:26.866 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:26.866 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:29:26.866 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:26.866 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:26.867 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:26.947 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-09-13T11:29:27.057 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2024-09-13T11:29:27.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2024-09-13T11:29:27.139 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 19 MB/s | 14 MB 00:00 2024-09-13T11:29:27.181 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-13T11:29:27.223 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-09-13T11:29:27.248 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 178 kB/s | 19 kB 00:00 2024-09-13T11:29:27.267 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:27.283 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 916 kB/s | 31 kB 00:00 2024-09-13T11:29:27.316 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.6 MB/s | 144 kB 00:00 2024-09-13T11:29:27.358 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-13T11:29:27.416 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.7 MB/s | 2.0 MB 00:00 2024-09-13T11:29:27.421 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:27.421 INFO:teuthology.orchestra.run.smithi118.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-13T11:29:27.468 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:29:27.487 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:29:27.487 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:29:27.513 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:29:27.513 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:27.513 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:29:27.513 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:27.513 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:27.513 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:27.642 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:27.708 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:27.731 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:27.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T11:29:27.774 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:27.775 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:29:27.775 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:27.791 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:29:27.791 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:29:27.803 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:27.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:29:27.846 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:27.847 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:29:27.847 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:27.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:29:28.343 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:28.387 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:29:28.404 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:28.475 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:29:28.482 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:29:28.483 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:28.484 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 145 k 2024-09-13T11:29:28.484 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 432 k 2024-09-13T11:29:28.484 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:29:28.494 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:02 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:29:28.537 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:28.538 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:28.538 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:29:28.538 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:29:28.539 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:29:28.540 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:29:28.540 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:29:28.540 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:28.540 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:28.540 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:28.540 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-09-13T11:29:28.540 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:28.541 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 19 M 2024-09-13T11:29:28.541 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 159 M 2024-09-13T11:29:28.541 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:29:28.596 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:29:28.627 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:28.628 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:28.628 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:29:28.628 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:28.628 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:29:28.628 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:29:28.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:29:28.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:29:28.630 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:29:28.630 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:28.630 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:29:28.630 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:28.630 INFO:teuthology.orchestra.run.smithi154.stdout:Install 12 Packages 2024-09-13T11:29:28.630 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:28.631 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 19 M 2024-09-13T11:29:28.631 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 159 M 2024-09-13T11:29:28.632 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:29:28.694 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:29:28.709 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 644 kB/s | 145 kB 00:00 2024-09-13T11:29:28.710 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:28.710 INFO:teuthology.orchestra.run.smithi038.stdout:Total 640 kB/s | 145 kB 00:00 2024-09-13T11:29:28.710 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:29:28.719 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:29:28.719 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:29:28.780 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:29:28.780 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:29:28.829 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:29:28.923 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4697.gd58 112 kB/s | 16 kB 00:00 2024-09-13T11:29:28.932 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:29:28.933 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:29:28.949 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4697.gd5 199 kB/s | 34 kB 00:00 2024-09-13T11:29:28.994 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:29.001 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:29:29.021 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:29.021 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:29:29.021 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T11:29:29.021 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:29.029 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:29.062 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:29:29.074 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-09-13T11:29:29.133 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-09-13T11:29:29.191 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T11:29:29.208 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-13T11:29:29.275 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-13T11:29:29.279 INFO:teuthology.orchestra.run.smithi154.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 184 kB/s | 34 kB 00:00 2024-09-13T11:29:29.301 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-13T11:29:29.305 INFO:teuthology.orchestra.run.smithi154.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 78 kB/s | 16 kB 00:00 2024-09-13T11:29:29.317 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-13T11:29:29.343 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-13T11:29:29.472 INFO:teuthology.orchestra.run.smithi154.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-13T11:29:29.485 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 20 MB/s | 14 MB 00:00 2024-09-13T11:29:29.514 INFO:teuthology.orchestra.run.smithi154.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-09-13T11:29:29.552 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.8 MB/s | 863 kB 00:00 2024-09-13T11:29:29.555 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:29.555 INFO:teuthology.orchestra.run.smithi204.stdout:Total 19 MB/s | 19 MB 00:01 2024-09-13T11:29:29.564 INFO:teuthology.orchestra.run.smithi154.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:29:29.590 INFO:teuthology.orchestra.run.smithi154.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-13T11:29:29.602 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:29:29.621 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:29:29.622 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:29:29.773 INFO:teuthology.orchestra.run.smithi154.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 21 MB/s | 14 MB 00:00 2024-09-13T11:29:29.815 INFO:teuthology.orchestra.run.smithi154.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-09-13T11:29:29.849 INFO:teuthology.orchestra.run.smithi154.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-09-13T11:29:29.890 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:29:29.890 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:29:29.891 INFO:teuthology.orchestra.run.smithi154.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.8 MB/s | 2.0 MB 00:00 2024-09-13T11:29:29.915 INFO:teuthology.orchestra.run.smithi154.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 312 kB/s | 31 kB 00:00 2024-09-13T11:29:29.941 INFO:teuthology.orchestra.run.smithi154.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-09-13T11:29:29.944 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:29.944 INFO:teuthology.orchestra.run.smithi154.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-13T11:29:29.992 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:29:30.011 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:29:30.011 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:29:30.033 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T11:29:30.033 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:30.033 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:29:30.034 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:29:30.034 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:30.034 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:30.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-09-13T11:29:30.270 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:29:30.270 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:29:30.477 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:30.552 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:29:30.587 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:29:30.634 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:29:30.665 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:29:30.764 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:29:30.842 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:30.891 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:29:30.904 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:29:30.935 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:29:30.980 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:30.981 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:29:30.981 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:30.985 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:29:30.992 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:29:31.035 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:29:31.045 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:29:31.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T11:29:31.081 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:29:31.081 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:31.112 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:29:31.114 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:29:31.219 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:29:31.354 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:29:31.449 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:29:31.508 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:29:31.545 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:31.577 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:29:31.640 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:01 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:31.779 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:31.780 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T11:29:31.781 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:31.782 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 19 M 2024-09-13T11:29:31.783 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 159 M 2024-09-13T11:29:31.783 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:29:32.221 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:29:32.301 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:32.388 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4697.gd58 112 kB/s | 16 kB 00:00 2024-09-13T11:29:32.405 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4697.gd5 209 kB/s | 34 kB 00:00 2024-09-13T11:29:32.564 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-13T11:29:32.639 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-13T11:29:32.698 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T11:29:32.723 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 786 kB/s | 19 kB 00:00 2024-09-13T11:29:32.798 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-13T11:29:32.840 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.1 MB/s | 863 kB 00:00 2024-09-13T11:29:32.890 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-09-13T11:29:32.924 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.2 MB/s | 188 kB 00:00 2024-09-13T11:29:32.958 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 263 kB/s | 31 kB 00:00 2024-09-13T11:29:33.125 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4697.gd58f73 16 MB/s | 14 MB 00:00 2024-09-13T11:29:33.129 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:33.129 INFO:teuthology.orchestra.run.smithi038.stdout:Total 14 MB/s | 19 MB 00:01 2024-09-13T11:29:33.181 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:29:33.202 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:29:33.203 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:29:33.496 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:29:33.497 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:29:34.110 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:29:34.201 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T11:29:34.244 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T11:29:34.280 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:29:34.293 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T11:29:34.321 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T11:29:34.335 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:34.418 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T11:29:34.555 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:34.555 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:29:34.555 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:29:34.555 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:29:34.556 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:29:34.556 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:29:34.556 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:29:34.556 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:29:34.556 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:29:34.556 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:34.557 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:29:34.557 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:29:34.557 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T11:29:34.646 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T11:29:34.702 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T11:29:34.729 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:34.758 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:29:34.761 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T11:29:34.815 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:29:34.891 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:29:34.892 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:29:34.892 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:29:34.892 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:29:34.892 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:29:34.892 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:34.892 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:35.154 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:29:35.713 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:10 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:35.846 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:35.847 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-09-13T11:29:35.847 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:29:35.847 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:29:35.848 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:29:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:29:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:35.849 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:29:35.849 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-09-13T11:29:35.849 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-09-13T11:29:35.849 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:35.849 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 38 M 2024-09-13T11:29:35.850 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 212 M 2024-09-13T11:29:35.850 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:29:36.363 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:29:36.364 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:36.364 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:29:36.364 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:29:36.695 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:29:36.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:29:36.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:29:36.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:29:36.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:29:36.696 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:29:36.696 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:36.696 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:36.817 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:36.817 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:29:36.817 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:29:36.817 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:29:36.817 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:29:36.817 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:29:36.817 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:29:36.818 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:29:36.818 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:29:36.818 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:36.818 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:29:36.818 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:29:36.866 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-46 15 MB/s | 7.4 MB 00:00 2024-09-13T11:29:36.908 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 349 kB/s | 191 kB 00:00 2024-09-13T11:29:36.974 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:29:37.067 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 163 kB/s | 33 kB 00:00 2024-09-13T11:29:37.123 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:29:37.123 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:37.123 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:29:37.123 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:29:37.124 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:29:37.125 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:29:37.125 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:37.125 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:37.134 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-09-13T11:29:37.184 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): libgfortran-11.5.0-2.el9.x86_64.rpm 973 kB/s | 801 kB 00:00 2024-09-13T11:29:37.209 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 511 kB/s | 38 kB 00:00 2024-09-13T11:29:37.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:29:37.486 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 887 kB/s | 245 kB 00:00 2024-09-13T11:29:37.549 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:09 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:37.692 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:37.693 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-13T11:29:37.693 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:29:37.694 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================= 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-09-13T11:29:37.695 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:37.696 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 38 M 2024-09-13T11:29:37.696 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 212 M 2024-09-13T11:29:37.696 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:37.925 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:11 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:38.059 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:38.060 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:29:38.061 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout:======================================================================================= 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout:Install 12 Packages 2024-09-13T11:29:38.062 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:38.063 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 38 M 2024-09-13T11:29:38.063 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 212 M 2024-09-13T11:29:38.063 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:29:38.135 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-13T11:29:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T11:29:38.169 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-09-13T11:29:38.202 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-13T11:29:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:38.244 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-13T11:29:38.294 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 250 kB 00:00 2024-09-13T11:29:38.302 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-09-13T11:29:38.353 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-13T11:29:38.503 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 15 MB/s | 7.4 MB 00:00 2024-09-13T11:29:38.514 INFO:teuthology.orchestra.run.smithi154.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-13T11:29:38.581 INFO:teuthology.orchestra.run.smithi154.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-13T11:29:38.681 INFO:teuthology.orchestra.run.smithi154.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 196 kB/s | 33 kB 00:00 2024-09-13T11:29:38.712 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-13T11:29:38.740 INFO:teuthology.orchestra.run.smithi154.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 312 kB/s | 18 kB 00:00 2024-09-13T11:29:38.823 INFO:teuthology.orchestra.run.smithi154.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 461 kB/s | 38 kB 00:00 2024-09-13T11:29:38.853 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.0 MB/s | 4.9 MB 00:01 2024-09-13T11:29:38.974 INFO:teuthology.orchestra.run.smithi154.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 12 MB/s | 7.4 MB 00:00 2024-09-13T11:29:39.183 INFO:teuthology.orchestra.run.smithi154.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-13T11:29:39.362 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.7 MB/s | 5.1 MB 00:01 2024-09-13T11:29:39.508 INFO:teuthology.orchestra.run.smithi154.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2024-09-13T11:29:39.600 INFO:teuthology.orchestra.run.smithi154.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-13T11:29:39.920 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.7 MB/s | 3.0 MB 00:01 2024-09-13T11:29:39.995 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 3.0 MB/s | 4.9 MB 00:01 2024-09-13T11:29:40.162 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.0 MB/s | 250 kB 00:00 2024-09-13T11:29:40.196 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T11:29:40.196 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T11:29:40.196 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T11:29:40.197 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T11:29:40.208 INFO:teuthology.orchestra.run.smithi154.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.6 MB/s | 4.9 MB 00:01 2024-09-13T11:29:40.513 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.8 MB/s | 5.1 MB 00:01 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:40.532 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:40.533 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:40.600 INFO:teuthology.orchestra.run.smithi154.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 5.1 MB 00:01 2024-09-13T11:29:40.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T11:29:41.030 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:01 2024-09-13T11:29:41.030 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:41.031 INFO:teuthology.orchestra.run.smithi204.stdout:Total 11 MB/s | 38 MB 00:03 2024-09-13T11:29:41.370 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:11 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:41.388 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:29:41.415 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:29:41.415 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:29:41.509 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:41.510 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-09-13T11:29:41.510 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:29:41.510 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-09-13T11:29:41.510 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:29:41.510 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T11:29:41.510 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:29:41.510 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:29:41.511 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-09-13T11:29:41.512 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-09-13T11:29:41.512 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:41.512 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 38 M 2024-09-13T11:29:41.512 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 212 M 2024-09-13T11:29:41.512 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:29:41.857 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:29:41.858 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:29:42.025 INFO:teuthology.orchestra.run.smithi154.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.7 MB/s | 16 MB 00:02 2024-09-13T11:29:42.026 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:42.027 INFO:teuthology.orchestra.run.smithi154.stdout:Total 9.6 MB/s | 38 MB 00:03 2024-09-13T11:29:42.190 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-13T11:29:42.223 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.8 MB/s | 801 kB 00:00 2024-09-13T11:29:42.274 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-13T11:29:42.316 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-13T11:29:42.357 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 933 kB/s | 38 kB 00:00 2024-09-13T11:29:42.384 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:29:42.410 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:29:42.410 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:29:42.558 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 14 MB/s | 7.4 MB 00:00 2024-09-13T11:29:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:42.675 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-09-13T11:29:42.678 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:29:42.720 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:42.822 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:42.854 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:29:42.865 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:29:42.866 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:29:42.983 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-09-13T11:29:43.049 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:29:43.075 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-13T11:29:43.078 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:29:43.165 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:29:43.624 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:29:43.625 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.9 MB/s | 4.9 MB 00:01 2024-09-13T11:29:43.694 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:29:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:43.840 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:29:43.883 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:29:43.909 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-09-13T11:29:43.951 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:29:44.068 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:29:44.107 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:29:44.192 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:29:44.287 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.7 MB/s | 16 MB 00:05 2024-09-13T11:29:44.287 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:44.287 INFO:teuthology.orchestra.run.smithi118.stdout:Total 4.5 MB/s | 38 MB 00:08 2024-09-13T11:29:44.647 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:29:44.673 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:29:44.673 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:29:44.877 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:29:44.988 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:29:45.134 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:29:45.134 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:29:45.198 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:45.890 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:29:45.969 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:29:46.009 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:46.153 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:29:46.233 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:46.240 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:29:46.262 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:29:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:29:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:29:46.684 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.5 MB/s | 16 MB 00:03 2024-09-13T11:29:46.685 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:46.685 INFO:teuthology.orchestra.run.smithi038.stdout:Total 7.4 MB/s | 38 MB 00:05 2024-09-13T11:29:47.050 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:29:47.077 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:29:47.077 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:29:47.123 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:29:47.256 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:29:47.277 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:29:47.306 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:47.533 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:29:47.533 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:29:48.326 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:29:48.396 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T11:29:48.441 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:48.490 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:48.546 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:48.583 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T11:29:48.777 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T11:29:48.808 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T11:29:48.887 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T11:29:49.096 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:29:49.097 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:49.098 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:29:49.443 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:29:49.443 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:49.444 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:29:49.445 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:49.445 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:49.552 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:29:49.594 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T11:29:49.594 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T11:29:49.828 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:29:50.106 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:50.106 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:29:50.106 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:50.107 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:29:50.402 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:22 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:29:50.523 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:29:50.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:29:50.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:50.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:50.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:29:50.524 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:50.524 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:50.537 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:50.540 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:50.540 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:29:50.540 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:50.540 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:50.540 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:29:50.540 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:29:50.541 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:29:50.541 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:29:50.541 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:29:50.541 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:29:50.541 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:29:50.541 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:29:50.541 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:50.542 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.2 M 2024-09-13T11:29:50.543 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-09-13T11:29:50.543 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:50.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:29:51.017 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:51.079 INFO:teuthology.orchestra.run.smithi204.stdout:(1/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 306 kB/s | 49 kB 00:00 2024-09-13T11:29:51.204 INFO:teuthology.orchestra.run.smithi204.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 89 kB/s | 26 kB 00:00 2024-09-13T11:29:51.221 INFO:teuthology.orchestra.run.smithi204.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 63 kB/s | 19 kB 00:00 2024-09-13T11:29:51.562 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:25 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:51.597 INFO:teuthology.orchestra.run.smithi204.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-13T11:29:51.623 INFO:teuthology.orchestra.run.smithi204.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 409 kB/s | 222 kB 00:00 2024-09-13T11:29:51.664 INFO:teuthology.orchestra.run.smithi204.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 775 kB/s | 32 kB 00:00 2024-09-13T11:29:51.689 INFO:teuthology.orchestra.run.smithi204.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 596 kB/s | 279 kB 00:00 2024-09-13T11:29:51.696 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:51.697 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:51.697 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:29:51.697 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:51.697 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:29:51.697 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:29:51.697 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:29:51.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:29:51.698 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:51.699 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2024-09-13T11:29:51.699 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:51.699 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.2 M 2024-09-13T11:29:51.699 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 26 M 2024-09-13T11:29:51.699 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:29:51.716 INFO:teuthology.orchestra.run.smithi204.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 280 kB/s | 14 kB 00:00 2024-09-13T11:29:51.748 INFO:teuthology.orchestra.run.smithi204.stdout:(9/13): python3-requests-oauthlib-1.3.0-12.el9. 355 kB/s | 54 kB 00:00 2024-09-13T11:29:51.790 INFO:teuthology.orchestra.run.smithi204.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.1 MB/s | 220 kB 00:00 2024-09-13T11:29:51.815 INFO:teuthology.orchestra.run.smithi204.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 901 kB/s | 59 kB 00:00 2024-09-13T11:29:51.841 INFO:teuthology.orchestra.run.smithi204.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-13T11:29:51.874 INFO:teuthology.orchestra.run.smithi204.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-13T11:29:51.876 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:51.877 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-13T11:29:51.941 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:29:51.957 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:29:51.957 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:29:51.981 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 209 kB/s | 19 kB 00:00 2024-09-13T11:29:52.031 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 26 kB 00:00 2024-09-13T11:29:52.045 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:29:52.047 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:29:52.065 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 280 kB/s | 49 kB 00:00 2024-09-13T11:29:52.090 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T11:29:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T11:29:52.141 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-13T11:29:52.148 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:52.166 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 714 kB/s | 54 kB 00:00 2024-09-13T11:29:52.208 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-13T11:29:52.233 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-09-13T11:29:52.258 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-13T11:29:52.292 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-13T11:29:52.317 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-13T11:29:52.342 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-09-13T11:29:52.384 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-13T11:29:52.387 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:52.387 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-13T11:29:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:29:52.465 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:29:52.465 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:29:52.556 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:29:52.556 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:29:52.562 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:52.738 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:29:52.799 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:52.863 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:29:52.864 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:29:52.901 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:29:52.942 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:29:53.052 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:29:53.064 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:29:53.156 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:29:53.211 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:29:53.232 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:29:53.285 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:29:53.297 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:29:53.298 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:53.299 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:53.299 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:29:53.299 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:53.299 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:53.340 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:29:53.378 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:29:53.419 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:29:53.530 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:29:53.640 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:29:53.674 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:29:53.689 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:29:53.918 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:29:53.967 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:29:54.001 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:29:54.049 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:29:54.071 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:29:54.255 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:29:54.387 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:29:54.388 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:54.388 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:29:54.388 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:54.388 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:29:54.388 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:29:54.388 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:29:54.389 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:29:54.390 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:54.390 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:29:54.390 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:29:54.390 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2024-09-13T11:29:54.390 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:54.390 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.2 M 2024-09-13T11:29:54.391 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2024-09-13T11:29:54.391 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:29:54.409 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:29:54.460 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:29:54.493 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:29:54.541 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:29:54.563 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:29:54.815 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:29:54.816 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:29:54.817 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:29:54.817 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T11:29:54.818 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T11:29:54.819 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T11:29:54.819 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T11:29:55.121 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 297 kB/s | 26 kB 00:00 2024-09-13T11:29:55.144 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:29:55.144 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:55.144 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:55.144 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:29:55.145 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:55.146 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:55.155 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 160 kB/s | 19 kB 00:00 2024-09-13T11:29:55.214 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T11:29:55.214 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T11:29:55.215 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T11:29:55.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:55.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T11:29:55.216 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T11:29:55.216 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:55.216 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:29:55.264 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 215 kB/s | 49 kB 00:00 2024-09-13T11:29:55.289 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-13T11:29:55.322 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-13T11:29:55.348 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 913 kB/s | 54 kB 00:00 2024-09-13T11:29:55.364 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:29:55.365 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:29:55.390 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-13T11:29:55.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:29:55.415 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-13T11:29:55.440 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-13T11:29:55.466 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-13T11:29:55.491 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-13T11:29:55.516 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-13T11:29:55.550 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-13T11:29:55.552 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:55.552 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-13T11:29:55.624 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:29:55.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-09-13T11:29:55.639 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:29:55.639 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:29:55.696 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:55.697 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:29:55.728 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:29:55.729 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:29:55.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:29:55.964 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:56.098 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:56.099 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:29:56.099 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:56.099 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:56.099 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:29:56.099 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-09-13T11:29:56.100 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:56.101 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 7.0 M 2024-09-13T11:29:56.101 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 33 M 2024-09-13T11:29:56.101 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:56.225 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:26 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:29:56.344 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.6 MB/s | 248 kB 00:00 2024-09-13T11:29:56.345 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:29:56.365 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:29:56.366 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:56.366 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:29:56.366 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:56.367 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:29:56.367 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T11:29:56.367 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:29:56.367 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T11:29:56.367 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T11:29:56.367 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T11:29:56.368 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T11:29:56.368 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T11:29:56.368 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T11:29:56.368 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T11:29:56.368 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T11:29:56.369 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T11:29:56.369 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T11:29:56.369 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T11:29:56.369 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T11:29:56.369 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:56.370 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:29:56.370 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:29:56.370 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-09-13T11:29:56.370 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:29:56.371 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.2 M 2024-09-13T11:29:56.371 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2024-09-13T11:29:56.371 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:29:56.395 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-13T11:29:56.420 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-13T11:29:56.445 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-13T11:29:56.470 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 701 kB/s | 154 kB 00:00 2024-09-13T11:29:56.515 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:29:56.527 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:29:56.554 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-09-13T11:29:56.556 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:56.556 INFO:teuthology.orchestra.run.smithi204.stdout:Total 15 MB/s | 7.0 MB 00:00 2024-09-13T11:29:56.577 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:29:56.626 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:29:56.631 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:29:56.637 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:29:56.638 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:29:56.661 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:29:56.661 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:56.661 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:29:56.661 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:56.661 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:29:56.661 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-09-13T11:29:56.662 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:29:56.663 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 7.0 M 2024-09-13T11:29:56.663 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 33 M 2024-09-13T11:29:56.663 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:29:56.670 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:29:56.710 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:29:56.738 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:29:56.738 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:29:56.821 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:29:56.939 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:29:56.965 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 194 kB/s | 19 kB 00:00 2024-09-13T11:29:56.990 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 207 kB/s | 26 kB 00:00 2024-09-13T11:29:57.004 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:29:57.046 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:29:57.048 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 270 kB/s | 49 kB 00:00 2024-09-13T11:29:57.099 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-13T11:29:57.103 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 737 kB/s | 154 kB 00:00 2024-09-13T11:29:57.124 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-13T11:29:57.149 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T11:29:57.161 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-13T11:29:57.187 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-13T11:29:57.191 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-13T11:29:57.212 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-13T11:29:57.212 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:29:57.216 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-13T11:29:57.241 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-13T11:29:57.267 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-13T11:29:57.272 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:29:57.292 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-13T11:29:57.342 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-09-13T11:29:57.368 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 893 kB/s | 90 kB 00:00 2024-09-13T11:29:57.370 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:29:57.370 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-09-13T11:29:57.438 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:29:57.454 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:29:57.455 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:29:57.551 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:29:57.551 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:29:57.729 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 297 kB/s | 248 kB 00:00 2024-09-13T11:29:57.733 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:29:57.793 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:29:57.796 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:29:57.827 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:29:57.875 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:29:57.884 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:29:57.897 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:29:57.960 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:29:57.985 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:29:58.064 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:29:58.247 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T11:29:58.304 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T11:29:58.376 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T11:29:58.420 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T11:29:58.462 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T11:29:58.568 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T11:29:58.678 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T11:29:58.731 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T11:29:58.763 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:29:58.763 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:29:58.763 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:29:58.763 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:29:58.763 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:29:58.763 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:29:58.799 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:29:58.800 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:29:58.800 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:29:59.019 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:59.020 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:29:59.136 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:29:59.136 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:59.136 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:29:59.136 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:29:59.136 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:29:59.137 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:29:59.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2024-09-13T11:29:59.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:29:59.496 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T11:29:59.544 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T11:29:59.586 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T11:29:59.635 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:29:59.826 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:29:59.958 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:29:59.959 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:59.960 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:29:59.960 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:29:59.960 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-09-13T11:29:59.960 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:29:59.960 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 979 k 2024-09-13T11:29:59.960 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.9 M 2024-09-13T11:29:59.960 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:29:59.963 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:00.096 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:00.097 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:00.098 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:00.098 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-09-13T11:30:00.098 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:00.098 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 7.0 M 2024-09-13T11:30:00.098 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 33 M 2024-09-13T11:30:00.098 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:00.381 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 357 kB/s | 80 kB 00:00 2024-09-13T11:30:00.439 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.1 MB/s | 900 kB 00:00 2024-09-13T11:30:00.440 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:00.440 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.0 MB/s | 979 kB 00:00 2024-09-13T11:30:00.451 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:30:00.473 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:30:00.473 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:30:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T11:30:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T11:30:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T11:30:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T11:30:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T11:30:00.537 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.6 MB/s | 6.0 MB 00:03 2024-09-13T11:30:00.539 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:00.539 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.8 MB/s | 7.0 MB 00:03 2024-09-13T11:30:00.554 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:30:00.554 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:30:00.613 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:00.625 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:00.726 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:00.726 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:00.745 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:30:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T11:30:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T11:30:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T11:30:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T11:30:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:00.825 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:00.884 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:30:00.920 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-09-13T11:30:00.948 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 518 kB/s | 154 kB 00:00 2024-09-13T11:30:00.955 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:00.970 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 770 kB/s | 248 kB 00:00 2024-09-13T11:30:01.021 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 768 kB/s | 58 kB 00:00 2024-09-13T11:30:01.041 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:01.046 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-13T11:30:01.068 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T11:30:01.080 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-09-13T11:30:01.083 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:01.083 INFO:teuthology.orchestra.run.smithi118.stdout:Total 7.1 MB/s | 7.0 MB 00:00 2024-09-13T11:30:01.156 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:01.168 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:01.168 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:01.210 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:30:01.263 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:30:01.267 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:01.269 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:01.572 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:01.666 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:01.674 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:01.674 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:30:01.754 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:30:01.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:30:01.806 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T11:30:01.807 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-09-13T11:30:01.808 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:01.809 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 7.0 M 2024-09-13T11:30:01.809 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 33 M 2024-09-13T11:30:01.809 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:01.832 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:30:01.866 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:30:01.934 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:30:01.942 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:30:01.942 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:01.942 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:30:01.942 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:30:01.942 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:01.942 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:01.959 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:30:02.194 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-volume 2024-09-13T11:30:02.319 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 720 kB/s | 154 kB 00:00 2024-09-13T11:30:02.352 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:30:02.353 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-13T11:30:02.404 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-13T11:30:02.429 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 766 kB/s | 58 kB 00:00 2024-09-13T11:30:02.454 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-13T11:30:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:30:02.537 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:30:02.561 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:30:02.755 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:30:02.755 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:30:02.755 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:30:02.755 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:30:02.755 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:30:02.755 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:30:02.778 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:02.912 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:02.913 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:02.913 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-09-13T11:30:02.913 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:02.913 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:30:02.913 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:30:02.913 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:02.914 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:30:02.914 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:02.914 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:30:02.914 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:02.914 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 285 k 2024-09-13T11:30:02.914 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.3 M 2024-09-13T11:30:02.914 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:03.045 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:03.144 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T11:30:03.145 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:03.145 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T11:30:03.145 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:30:03.154 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:30:03.154 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:30:03.179 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:30:03.180 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:30:03.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2024-09-13T11:30:03.382 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:30:03.382 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:30:03.382 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:30:03.382 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:30:03.383 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:30:03.383 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:30:03.406 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:30:03.471 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:03.487 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:03.487 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:03.488 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:30:03.488 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:03.488 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.3 MB/s | 6.0 MB 00:01 2024-09-13T11:30:03.490 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:03.490 INFO:teuthology.orchestra.run.smithi038.stdout:Total 4.2 MB/s | 7.0 MB 00:01 2024-09-13T11:30:03.564 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:03.576 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:03.576 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:03.668 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:30:03.668 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:03.668 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:03.669 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:03.890 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:03.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2024-09-13T11:30:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:04.025 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:30:04.026 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:04.026 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2024-09-13T11:30:04.026 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:04.026 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 979 k 2024-09-13T11:30:04.026 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.9 M 2024-09-13T11:30:04.026 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:30:04.164 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 678 kB/s | 80 kB 00:00 2024-09-13T11:30:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T11:30:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T11:30:04.323 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-13T11:30:04.323 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:04.323 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.2 MB/s | 979 kB 00:00 2024-09-13T11:30:04.335 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:04.356 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:04.356 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:04.438 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:04.439 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:04.469 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:30:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:04.470 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:04.512 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:04.636 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:04.645 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:04.645 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-09-13T11:30:04.646 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:04.647 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 979 k 2024-09-13T11:30:04.647 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 2.9 M 2024-09-13T11:30:04.647 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:04.749 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2024-09-13T11:30:04.775 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:30:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T11:30:04.846 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:04.879 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T11:30:04.954 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T11:30:04.985 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:30:05.099 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 517 kB/s | 80 kB 00:00 2024-09-13T11:30:05.258 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 2.8 MB/s | 900 kB 00:00 2024-09-13T11:30:05.259 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:05.259 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.6 MB/s | 979 kB 00:00 2024-09-13T11:30:05.270 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:05.292 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:05.293 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:05.324 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:05.374 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:05.375 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:05.462 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:05.463 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 127 k 2024-09-13T11:30:05.464 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 456 k 2024-09-13T11:30:05.464 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:30:05.560 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:05.561 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:30:05.591 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:05.642 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 710 kB/s | 127 kB 00:00 2024-09-13T11:30:05.642 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:05.643 INFO:teuthology.orchestra.run.smithi204.stdout:Total 705 kB/s | 127 kB 00:00 2024-09-13T11:30:05.643 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:30:05.648 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:30:05.648 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:30:05.712 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:30:05.712 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:30:05.718 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:30:05.754 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T11:30:05.754 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T11:30:05.754 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T11:30:05.754 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T11:30:05.754 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T11:30:05.754 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T11:30:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:05.845 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:30:05.845 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:05.845 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:05.845 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:30:05.845 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:05.845 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:05.861 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:30:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:06.039 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T11:30:06.039 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:06.039 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:06.040 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:06.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-volume 2024-09-13T11:30:06.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-09-13T11:30:06.399 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:06.436 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:06.436 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:30:06.649 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:06.649 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:06.649 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:30:06.649 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:06.649 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:06.649 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:06.711 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:30:06.711 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:06.711 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:06.711 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:30:06.711 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:06.711 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:06.730 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:06.863 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:06.863 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 285 k 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 1.3 M 2024-09-13T11:30:06.864 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:30:06.894 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:06.910 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2024-09-13T11:30:07.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-volume 2024-09-13T11:30:07.034 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:07.034 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:07.034 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-09-13T11:30:07.035 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:07.036 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 979 k 2024-09-13T11:30:07.037 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.9 M 2024-09-13T11:30:07.037 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:07.068 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-13T11:30:07.069 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:07.069 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.4 MB/s | 285 kB 00:00 2024-09-13T11:30:07.069 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:07.077 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:07.077 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:07.122 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:07.123 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:07.207 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 541 kB/s | 80 kB 00:00 2024-09-13T11:30:07.332 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-13T11:30:07.332 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:07.333 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.2 MB/s | 979 kB 00:00 2024-09-13T11:30:07.345 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:07.353 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:07.368 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:07.368 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:07.436 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:07.453 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:07.453 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:07.453 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:30:07.453 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:07.454 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:07.454 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:07.509 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:07.578 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:07.607 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:07.647 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:07.655 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:07.656 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:30:07.656 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:07.714 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:07.715 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:07.715 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-09-13T11:30:07.715 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:07.715 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 285 k 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 1.3 M 2024-09-13T11:30:07.716 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:07.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2024-09-13T11:30:07.806 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T11:30:07.875 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:07.940 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T11:30:07.940 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:07.940 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T11:30:07.941 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:07.949 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:07.949 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:07.983 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:07.983 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:08.203 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:08.266 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:08.282 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:08.282 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:08.283 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:30:08.283 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:08.319 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:08.459 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:08.460 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 32 k 2024-09-13T11:30:08.461 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 142 k 2024-09-13T11:30:08.461 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:30:08.576 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:08.576 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:08.576 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:08.576 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:08.576 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:08.576 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:08.585 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T11:30:08.585 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T11:30:08.605 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 221 kB/s | 32 kB 00:00 2024-09-13T11:30:08.605 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:08.605 INFO:teuthology.orchestra.run.smithi204.stdout:Total 218 kB/s | 32 kB 00:00 2024-09-13T11:30:08.606 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:30:08.609 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:30:08.609 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:30:08.631 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:30:08.632 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:30:08.718 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:30:08.819 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:08.863 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T11:30:08.863 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:08.863 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:08.864 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T11:30:08.864 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:08.864 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:08.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2024-09-13T11:30:09.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-volume 2024-09-13T11:30:09.221 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:09.437 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:09.437 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:09.437 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:09.437 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:09.437 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:09.437 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:09.467 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:09.467 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:09.467 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:30:09.467 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:09.468 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:09.468 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:09.479 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:09.617 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:30:09.618 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:09.619 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 127 k 2024-09-13T11:30:09.619 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 456 k 2024-09-13T11:30:09.619 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:30:09.646 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2024-09-13T11:30:09.787 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2024-09-13T11:30:09.788 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:09.833 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 593 kB/s | 127 kB 00:00 2024-09-13T11:30:09.833 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:09.834 INFO:teuthology.orchestra.run.smithi154.stdout:Total 589 kB/s | 127 kB 00:00 2024-09-13T11:30:09.834 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:09.840 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:09.840 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:09.903 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:09.904 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:09.927 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:09.928 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:09.928 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-09-13T11:30:09.928 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:09.928 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:09.928 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T11:30:09.928 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:09.929 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:09.929 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:09.929 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:30:09.929 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:09.929 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 285 k 2024-09-13T11:30:09.929 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.3 M 2024-09-13T11:30:09.929 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:10.058 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:10.137 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-13T11:30:10.137 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:10.138 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-13T11:30:10.138 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:10.147 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:10.147 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:10.156 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:10.180 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:10.180 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:10.211 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:10.302 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:10.345 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:10.346 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:30:10.346 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:10.372 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:10.412 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2024-09-13T11:30:10.420 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:10.489 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:10.506 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:10.506 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:10.506 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T11:30:10.506 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:10.508 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:10.509 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:10.510 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:30:10.510 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:10.510 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 127 k 2024-09-13T11:30:10.510 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 456 k 2024-09-13T11:30:10.510 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:10.633 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:10.721 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 599 kB/s | 127 kB 00:00 2024-09-13T11:30:10.722 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:10.722 INFO:teuthology.orchestra.run.smithi118.stdout:Total 595 kB/s | 127 kB 00:00 2024-09-13T11:30:10.722 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:10.728 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:10.728 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:10.791 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:10.792 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:10.917 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:10.918 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:10.918 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:10.918 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:10.938 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:10.978 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:11.052 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:11.069 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:11.113 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:11.114 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:30:11.115 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:11.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2024-09-13T11:30:11.181 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2024-09-13T11:30:11.510 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T11:30:11.511 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:11.511 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:11.511 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T11:30:11.511 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:11.511 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:11.570 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:11.744 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:11.746 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:11.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-09-13T11:30:11.833 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:11.837 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:11.851 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:11.851 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:11.851 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:11.851 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:11.851 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:11.851 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:11.877 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:11.878 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:30:11.878 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:11.881 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:11.883 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:30:11.883 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:11.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2024-09-13T11:30:11.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2024-09-13T11:30:12.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2024-09-13T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:12.510 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:12.566 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:12.566 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:12.566 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 127 k 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 456 k 2024-09-13T11:30:12.567 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:12.573 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:12.600 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:12.643 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:12.644 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:30:12.644 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:12.695 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:12.706 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:12.706 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:12.706 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T11:30:12.706 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:12.706 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:30:12.707 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T11:30:12.707 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:12.707 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:30:12.707 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:12.707 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:30:12.707 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:12.708 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 32 k 2024-09-13T11:30:12.708 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 142 k 2024-09-13T11:30:12.708 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:30:12.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2024-09-13T11:30:12.750 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 695 kB/s | 127 kB 00:00 2024-09-13T11:30:12.751 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:12.751 INFO:teuthology.orchestra.run.smithi038.stdout:Total 689 kB/s | 127 kB 00:00 2024-09-13T11:30:12.751 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:12.757 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:12.757 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:12.787 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:12.825 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:12.825 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:12.832 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:12.833 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:30:12.834 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:12.861 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 206 kB/s | 32 kB 00:00 2024-09-13T11:30:12.862 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:12.862 INFO:teuthology.orchestra.run.smithi154.stdout:Total 204 kB/s | 32 kB 00:00 2024-09-13T11:30:12.862 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:12.866 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:12.866 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:12.903 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:12.903 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:12.912 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2024-09-13T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:13.092 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:13.099 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:13.287 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:13.379 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:13.422 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:13.423 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:30:13.423 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:13.471 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:13.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2024-09-13T11:30:13.492 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:13.603 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:13.604 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 32 k 2024-09-13T11:30:13.605 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 142 k 2024-09-13T11:30:13.605 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:13.742 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:13.742 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:13.742 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:13.742 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:13.743 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:13.743 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:13.751 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-13T11:30:13.752 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:13.752 INFO:teuthology.orchestra.run.smithi118.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-13T11:30:13.752 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:13.755 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:13.755 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:13.760 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:13.760 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:13.760 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:13.760 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:13.760 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:13.760 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:13.770 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:13.770 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:13.862 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:13.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2024-09-13T11:30:13.973 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:14.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-09-13T11:30:14.054 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:14.143 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:14.186 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:14.187 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-09-13T11:30:14.187 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:14.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2024-09-13T11:30:14.363 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:14.499 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:14.589 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:14.608 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:14.608 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:14.608 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:14.608 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:14.608 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:14.608 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:14.624 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:14.631 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:14.632 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:30:14.632 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:14.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2024-09-13T11:30:14.720 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:14.766 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:30:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:14.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2024-09-13T11:30:14.826 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:14.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-09-13T11:30:14.960 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 85 k 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 231 k 2024-09-13T11:30:14.961 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:30:15.150 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 453 kB/s | 85 kB 00:00 2024-09-13T11:30:15.150 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:15.150 INFO:teuthology.orchestra.run.smithi204.stdout:Total 450 kB/s | 85 kB 00:00 2024-09-13T11:30:15.150 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:30:15.157 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:30:15.157 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:30:15.222 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:30:15.222 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:30:15.279 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:15.364 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:30:15.369 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:15.376 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:15.413 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:15.413 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:30:15.413 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:15.422 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:15.464 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:15.466 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:15.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2024-09-13T11:30:15.510 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:15.510 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:30:15.510 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:15.561 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:15.562 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:15.562 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-13T11:30:15.562 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:15.562 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 32 k 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 142 k 2024-09-13T11:30:15.563 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:15.611 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2024-09-13T11:30:15.712 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 213 kB/s | 32 kB 00:00 2024-09-13T11:30:15.712 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:15.713 INFO:teuthology.orchestra.run.smithi038.stdout:Total 211 kB/s | 32 kB 00:00 2024-09-13T11:30:15.713 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:15.716 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:15.717 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:15.735 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:15.736 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:15.876 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:15.937 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:16.070 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:16.122 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:16.122 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:16.123 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:30:16.123 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:16.123 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:16.123 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:16.159 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:16.173 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:16.203 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:16.204 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:30:16.204 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:16.263 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:16.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2024-09-13T11:30:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:16.307 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:16.308 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:30:16.308 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:16.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2024-09-13T11:30:16.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-09-13T11:30:16.549 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:16.549 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:16.549 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:16.550 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:16.550 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:16.550 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:16.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-09-13T11:30:16.844 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:16.933 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:16.945 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:16.974 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:16.975 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:16.976 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:30:16.977 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:17.034 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:17.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2024-09-13T11:30:17.077 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:17.078 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:30:17.078 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:17.111 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:17.112 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:30:17.113 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:17.113 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.2 M 2024-09-13T11:30:17.113 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 14 M 2024-09-13T11:30:17.113 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:30:17.169 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2024-09-13T11:30:17.326 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:17.419 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:17.453 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-13T11:30:17.453 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:17.454 INFO:teuthology.orchestra.run.smithi204.stdout:Total 9.3 MB/s | 3.2 MB 00:00 2024-09-13T11:30:17.454 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:30:17.463 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:30:17.463 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:30:17.465 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:17.466 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:30:17.466 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:17.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-09-13T11:30:17.550 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:30:17.550 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:30:17.623 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:17.713 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:17.728 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:17.758 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:17.759 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:30:17.759 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:17.818 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:17.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2024-09-13T11:30:17.861 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:17.862 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:30:17.862 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:17.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2024-09-13T11:30:18.019 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:30:18.087 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:18.112 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:18.113 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:18.113 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T11:30:18.113 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:18.113 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:18.113 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:18.130 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:18.223 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:18.269 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:18.270 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:30:18.270 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:18.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-09-13T11:30:18.419 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:18.503 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:18.510 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:18.554 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:18.555 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-13T11:30:18.555 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:18.593 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:18.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2024-09-13T11:30:18.637 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:18.638 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:30:18.638 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:18.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2024-09-13T11:30:18.935 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:19.028 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:19.074 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:19.075 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:30:19.075 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:19.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-09-13T11:30:19.216 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:19.246 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:19.246 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:19.246 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:30:19.246 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:19.246 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:19.246 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:19.313 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:19.350 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 85 k 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 231 k 2024-09-13T11:30:19.351 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:30:19.404 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:19.446 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:19.447 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-09-13T11:30:19.447 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:19.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2024-09-13T11:30:19.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2024-09-13T11:30:19.558 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 417 kB/s | 85 kB 00:00 2024-09-13T11:30:19.558 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:19.558 INFO:teuthology.orchestra.run.smithi154.stdout:Total 414 kB/s | 85 kB 00:00 2024-09-13T11:30:19.558 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:19.564 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:19.564 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:19.629 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:19.630 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:19.740 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:19.777 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:19.832 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:19.879 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:19.880 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:30:19.880 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:19.887 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:19.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-09-13T11:30:20.078 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:28:28 AM UTC. 2024-09-13T11:30:20.117 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:20.214 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-09-13T11:30:20.215 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:20.216 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 170 k 2024-09-13T11:30:20.216 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 490 k 2024-09-13T11:30:20.216 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-09-13T11:30:20.251 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:20.251 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:20.251 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:20.251 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 85 k 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 231 k 2024-09-13T11:30:20.252 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:20.261 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:20.424 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 820 kB/s | 170 kB 00:00 2024-09-13T11:30:20.424 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:20.425 INFO:teuthology.orchestra.run.smithi204.stdout:Total 815 kB/s | 170 kB 00:00 2024-09-13T11:30:20.425 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-09-13T11:30:20.432 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-09-13T11:30:20.432 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-09-13T11:30:20.471 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 390 kB/s | 85 kB 00:00 2024-09-13T11:30:20.471 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:20.472 INFO:teuthology.orchestra.run.smithi118.stdout:Total 387 kB/s | 85 kB 00:00 2024-09-13T11:30:20.472 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:20.479 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:20.479 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:20.497 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-09-13T11:30:20.497 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-09-13T11:30:20.541 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:20.542 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:20.542 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:20.545 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:20.545 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:20.545 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:20.545 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:20.545 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:20.545 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:20.635 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:20.658 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-09-13T11:30:20.681 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:20.682 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:30:20.682 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:20.692 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:20.755 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:20.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-09-13T11:30:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:20.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2024-09-13T11:30:21.175 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:21.188 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:51 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:21.403 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:55 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T11:30:21.441 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:21.441 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:21.441 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-09-13T11:30:21.441 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:21.441 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:21.441 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-09-13T11:30:21.458 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:21.458 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:21.459 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:21.459 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:21.459 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:21.459 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:21.486 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:21.487 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-09-13T11:30:21.487 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:21.541 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:21.542 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:30:21.543 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:21.543 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.2 M 2024-09-13T11:30:21.543 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 14 M 2024-09-13T11:30:21.543 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:30:21.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-09-13T11:30:21.672 DEBUG:teuthology.parallel:result is None 2024-09-13T11:30:21.757 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2024-09-13T11:30:21.910 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 8.6 MB/s | 3.2 MB 00:00 2024-09-13T11:30:21.910 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:21.911 INFO:teuthology.orchestra.run.smithi154.stdout:Total 8.6 MB/s | 3.2 MB 00:00 2024-09-13T11:30:21.911 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:21.920 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:21.920 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:22.008 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:22.008 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:22.151 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:52 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:22.290 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:22.291 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:22.291 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:22.291 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T11:30:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:22.292 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:22.292 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:22.292 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:30:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:22.292 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 85 k 2024-09-13T11:30:22.292 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 231 k 2024-09-13T11:30:22.292 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:22.339 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:57 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:22.460 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:22.465 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 493 kB/s | 85 kB 00:00 2024-09-13T11:30:22.466 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:22.466 INFO:teuthology.orchestra.run.smithi038.stdout:Total 489 kB/s | 85 kB 00:00 2024-09-13T11:30:22.466 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:22.473 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:22.473 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:22.476 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:22.476 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:30:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:22.478 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.2 M 2024-09-13T11:30:22.478 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 14 M 2024-09-13T11:30:22.479 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:22.510 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:22.536 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:22.536 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:22.536 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T11:30:22.536 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:22.536 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:22.536 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:22.540 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:22.540 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:22.688 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:22.790 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:22.828 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.1 MB/s | 3.2 MB 00:00 2024-09-13T11:30:22.829 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:22.829 INFO:teuthology.orchestra.run.smithi118.stdout:Total 9.1 MB/s | 3.2 MB 00:00 2024-09-13T11:30:22.829 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:22.836 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:22.836 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:22.922 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:22.922 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:23.187 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:23.443 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:23.443 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:23.443 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:23.443 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:23.443 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:23.443 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:23.485 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:23.485 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T11:30:23.485 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:23.485 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:23.486 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:23.666 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:23.666 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:23.666 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:23.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-09-13T11:30:23.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2024-09-13T11:30:24.327 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:54 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:30:24.467 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:24.468 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.2 M 2024-09-13T11:30:24.468 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 14 M 2024-09-13T11:30:24.468 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:24.480 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:58 ago on Fri 13 Sep 2024 11:28:26 AM UTC. 2024-09-13T11:30:24.618 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-13T11:30:24.619 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:24.620 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 170 k 2024-09-13T11:30:24.620 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 490 k 2024-09-13T11:30:24.620 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-13T11:30:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:24.650 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:24.650 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:24.810 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-13T11:30:24.811 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:24.811 INFO:teuthology.orchestra.run.smithi038.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-13T11:30:24.811 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:24.821 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:24.821 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:24.838 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 778 kB/s | 170 kB 00:00 2024-09-13T11:30:24.839 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:24.839 INFO:teuthology.orchestra.run.smithi154.stdout:Total 773 kB/s | 170 kB 00:00 2024-09-13T11:30:24.839 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-13T11:30:24.847 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-13T11:30:24.847 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-13T11:30:24.906 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2024-09-13T11:30:24.912 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-13T11:30:24.913 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-13T11:30:24.913 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:24.914 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-13T11:30:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:25.392 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:25.453 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:25.473 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:02:00 ago on Fri 13 Sep 2024 11:28:25 AM UTC. 2024-09-13T11:30:25.478 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:25.479 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T11:30:25.479 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T11:30:25.479 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:25.479 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T11:30:25.479 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:25.607 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-09-13T11:30:25.608 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:25.609 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 170 k 2024-09-13T11:30:25.609 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 490 k 2024-09-13T11:30:25.609 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-09-13T11:30:25.610 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:25.831 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 768 kB/s | 170 kB 00:00 2024-09-13T11:30:25.831 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:25.831 INFO:teuthology.orchestra.run.smithi118.stdout:Total 764 kB/s | 170 kB 00:00 2024-09-13T11:30:25.832 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-09-13T11:30:25.839 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-09-13T11:30:25.839 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-09-13T11:30:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:25.857 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-13T11:30:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:25.858 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-13T11:30:25.903 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-09-13T11:30:25.904 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-09-13T11:30:26.057 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-09-13T11:30:26.092 DEBUG:teuthology.parallel:result is None 2024-09-13T11:30:26.145 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:26.615 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:26.663 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:26.663 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:26.663 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:26.663 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:26.663 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:26.663 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:26.849 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:26.849 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:26.849 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-09-13T11:30:26.850 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:26.850 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:26.850 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-09-13T11:30:26.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-09-13T11:30:27.114 DEBUG:teuthology.parallel:result is None 2024-09-13T11:30:27.526 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:57 ago on Fri 13 Sep 2024 11:28:30 AM UTC. 2024-09-13T11:30:27.668 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-09-13T11:30:27.669 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:27.669 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-09-13T11:30:27.669 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:27.669 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-09-13T11:30:27.669 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 170 k 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 490 k 2024-09-13T11:30:27.670 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-09-13T11:30:27.881 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 809 kB/s | 170 kB 00:00 2024-09-13T11:30:27.881 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-09-13T11:30:27.882 INFO:teuthology.orchestra.run.smithi038.stdout:Total 802 kB/s | 170 kB 00:00 2024-09-13T11:30:27.882 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-09-13T11:30:27.891 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-09-13T11:30:27.891 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-09-13T11:30:27.961 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-09-13T11:30:27.961 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-09-13T11:30:28.123 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-09-13T11:30:28.231 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:28.682 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:28.955 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T11:30:28.956 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:28.956 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-09-13T11:30:28.956 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T11:30:28.956 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:28.956 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-09-13T11:30:29.190 DEBUG:teuthology.parallel:result is None 2024-09-13T11:30:29.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:30:29.334 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T11:30:29.357 INFO:teuthology.orchestra.run.smithi038.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.357 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.358 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T11:30:29.360 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:30:29.506 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T11:30:29.531 INFO:teuthology.orchestra.run.smithi118.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.532 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.532 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T11:30:29.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:30:29.672 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T11:30:29.698 INFO:teuthology.orchestra.run.smithi154.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.698 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.698 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T11:30:29.700 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T11:30:29.848 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T11:30:29.873 INFO:teuthology.orchestra.run.smithi204.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.873 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T11:30:29.874 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T11:30:29.876 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T11:30:29.876 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:29.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:30:29.909 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:30:29.909 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:30:29.940 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:30:29.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:30:29.972 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:30:29.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T11:30:30.003 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T11:30:30.003 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:30.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:30:30.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:30:30.104 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:30:30.104 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:30:30.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:30:30.203 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:30:30.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:30:30.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:30:30.302 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:30:30.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T11:30:30.332 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T11:30:30.401 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T11:30:30.402 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:30.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:30:30.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:30:30.503 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:30:30.504 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:30:30.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:30:30.601 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:30:30.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:30:30.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:30:30.699 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:30:30.699 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T11:30:30.729 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T11:30:30.798 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T11:30:30.799 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:30.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:30:30.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:30:30.901 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:30:30.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:30:30.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:30:30.999 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:30:31.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:30:31.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:30:31.098 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:30:31.098 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T11:30:31.128 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T11:30:31.197 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T11:30:31.285 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T11:30:31.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2024-09-13T11:30:31.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2024-09-13T11:30:31.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2024-09-13T11:30:31.293 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-09-13T11:30:31.320 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T11:30:31.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:30:31.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:30:31.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:30:31.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T11:30:31.392 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T11:30:31.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:30:31.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:30:31.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:30:31.435 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T11:30:31.476 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T11:30:31.477 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore odsync write': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 0.5}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T11:30:31.477 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421', 'branch': 'wip-aclamk-cbt-combined', 'description': 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/pggrow_host thrashosds-health workloads/dedup-io-mixed}', 'email': 'akupczyk@redhat.com', 'first_in_suite': False, 'job_id': '7903421', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore odsync write': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 0.5}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'owner': 'scheduled_akupczyk@teuthology', 'priority': 160, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 5394, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'sleep_before_teardown': 0, 'subset': '582/12160', 'suite': 'rados', 'suite_branch': 'wip-aclamk-cbt-combined', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'targets': {'smithi038.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIHXVysjpVe/7grEmhcU1oTH1zIv/vx+Pgw5HFraDCdmShzubyuI0kMjUsVH/xajsaE+f8dBouuglNZGG0D3pqY=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAB9yTrwh3eB8qVf+11IbFEQ+tBTYi2cDpSWXY/dlcYdV6hrHY3F6IlLCozJZfBEo+tYtKJ34pvZYm9rc2VwqUs=', 'smithi154.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCK+8vNd70wRg3y5L81N0Z/3GBvj4rvzoAse31fmhzYvm4f9RThrnjCTLU3CjdJJYpQnd4/CofalBb9uNPqiMYw=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD3fsQAGRtm6fT60TA+6AUzgeo2jfbutPDpZCruJccQwNekzsnOHIeDYNuls2/YL/B2kVQ+jJIY9657AKXZXSPY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore odsync write': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 0.5}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'thrashosds': {'chance_pgnum_grow': 2, 'chance_pgpnum_fix': 1, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1200}}, {'exec': {'client.0': ['sudo ceph osd pool create low_tier 4']}}, {'rados': {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'set_chunk': 30, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_09:37:56', 'tube': 'smithi', 'user': 'akupczyk', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T11:30:31.478 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:30:31.500 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:30:31.502 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:30:31.504 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T11:30:31.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:30:31.557 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:30:31.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:30:31.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T11:30:31.587 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:31.587 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:30:31.643 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:30:31.643 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-09-13T11:30:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:30:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:31.701 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T11:30:31.701 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:31.702 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:31.702 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-13 11:30:25.746630927 +0000 2024-09-13T11:30:31.702 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-13 11:27:46.768110083 +0000 2024-09-13T11:30:31.702 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-13 11:27:46.768110083 +0000 2024-09-13T11:30:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-13 11:27:46.768110083 +0000 2024-09-13T11:30:31.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:30:31.768 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-13T11:30:31.769 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-13T11:30:31.769 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000395358 s, 1.3 MB/s 2024-09-13T11:30:31.770 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:30:31.828 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-09-13T11:30:31.885 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:30:31.885 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:31.885 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T11:30:31.886 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:31.886 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:31.886 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-13 11:30:25.747630908 +0000 2024-09-13T11:30:31.886 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-13 11:27:47.117103466 +0000 2024-09-13T11:30:31.886 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-13 11:27:47.117103466 +0000 2024-09-13T11:30:31.886 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-13 11:27:47.117103466 +0000 2024-09-13T11:30:31.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:30:31.951 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-13T11:30:31.951 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-13T11:30:31.951 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000297572 s, 1.7 MB/s 2024-09-13T11:30:31.953 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:30:32.010 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-09-13T11:30:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:30:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:32.066 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-13T11:30:32.066 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:32.067 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:32.067 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-13 11:30:25.747630908 +0000 2024-09-13T11:30:32.067 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-13 11:27:47.452097114 +0000 2024-09-13T11:30:32.067 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-13 11:27:47.452097114 +0000 2024-09-13T11:30:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-13 11:27:47.452097114 +0000 2024-09-13T11:30:32.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:30:32.132 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-13T11:30:32.132 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-13T11:30:32.132 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000256821 s, 2.0 MB/s 2024-09-13T11:30:32.133 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:30:32.191 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-09-13T11:30:32.248 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:30:32.248 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:32.248 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-13T11:30:32.248 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:32.248 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:32.249 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-09-13 11:30:25.747630908 +0000 2024-09-13T11:30:32.249 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-09-13 11:27:47.792090668 +0000 2024-09-13T11:30:32.249 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-09-13 11:27:47.792090668 +0000 2024-09-13T11:30:32.249 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-09-13 11:27:47.792090668 +0000 2024-09-13T11:30:32.249 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:30:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-09-13T11:30:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-09-13T11:30:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00026941 s, 1.9 MB/s 2024-09-13T11:30:32.315 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:30:32.372 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:32.373 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:30:32.373 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:30:32.392 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:30:32.392 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-09-13T11:30:32.454 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:30:32.454 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:32.454 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T11:30:32.454 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:32.455 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:32.455 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-13 11:30:23.722490312 +0000 2024-09-13T11:30:32.455 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-13 11:27:45.285930079 +0000 2024-09-13T11:30:32.455 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-13 11:27:45.285930079 +0000 2024-09-13T11:30:32.455 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-13 11:27:45.285930079 +0000 2024-09-13T11:30:32.455 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:30:32.521 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-13T11:30:32.521 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-13T11:30:32.521 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000118203 s, 4.3 MB/s 2024-09-13T11:30:32.522 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:30:32.581 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-13 11:30:23.722490312 +0000 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-13 11:27:45.592923719 +0000 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-13 11:27:45.592923719 +0000 2024-09-13T11:30:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-13 11:27:45.592923719 +0000 2024-09-13T11:30:32.639 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:30:32.705 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-13T11:30:32.705 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-13T11:30:32.705 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000112189 s, 4.6 MB/s 2024-09-13T11:30:32.707 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:30:32.764 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-09-13T11:30:32.821 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:30:32.821 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:32.821 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-13T11:30:32.821 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:32.821 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:32.821 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-13 11:30:23.723490291 +0000 2024-09-13T11:30:32.821 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-13 11:27:45.912917090 +0000 2024-09-13T11:30:32.822 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-13 11:27:45.912917090 +0000 2024-09-13T11:30:32.822 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-13 11:27:45.912917090 +0000 2024-09-13T11:30:32.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:30:32.888 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-13T11:30:32.888 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-13T11:30:32.888 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000110105 s, 4.7 MB/s 2024-09-13T11:30:32.889 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:30:32.946 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-09-13T11:30:33.002 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:30:33.002 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-13T11:30:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-13 11:30:23.723490291 +0000 2024-09-13T11:30:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-13 11:27:46.239910315 +0000 2024-09-13T11:30:33.003 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-13 11:27:46.239910315 +0000 2024-09-13T11:30:33.003 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-09-13 11:27:46.239910315 +0000 2024-09-13T11:30:33.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:30:33.069 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-13T11:30:33.070 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-13T11:30:33.070 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 8.7254e-05 s, 5.9 MB/s 2024-09-13T11:30:33.070 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:30:33.126 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:33.126 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:30:33.126 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:30:33.146 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:30:33.146 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 11:30:22.795855607 +0000 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 11:27:45.523489512 +0000 2024-09-13T11:30:33.205 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 11:27:45.523489512 +0000 2024-09-13T11:30:33.206 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 11:27:45.523489512 +0000 2024-09-13T11:30:33.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:30:33.277 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T11:30:33.277 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T11:30:33.277 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000281399 s, 1.8 MB/s 2024-09-13T11:30:33.278 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:30:33.336 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 11:30:22.795855607 +0000 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 11:27:45.843480773 +0000 2024-09-13T11:30:33.394 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 11:27:45.843480773 +0000 2024-09-13T11:30:33.395 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 11:27:45.843480773 +0000 2024-09-13T11:30:33.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:30:33.461 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T11:30:33.461 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T11:30:33.462 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000222118 s, 2.3 MB/s 2024-09-13T11:30:33.463 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:30:33.520 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-09-13T11:30:33.577 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:30:33.577 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-13T11:30:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:33.577 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:33.578 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 11:30:22.795855607 +0000 2024-09-13T11:30:33.578 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 11:27:46.151472362 +0000 2024-09-13T11:30:33.578 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 11:27:46.151472362 +0000 2024-09-13T11:30:33.578 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 11:27:46.151472362 +0000 2024-09-13T11:30:33.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:30:33.643 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T11:30:33.643 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T11:30:33.643 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000195509 s, 2.6 MB/s 2024-09-13T11:30:33.645 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:30:33.701 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-09-13T11:30:33.757 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-13 11:30:22.796855580 +0000 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-13 11:27:46.481463350 +0000 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-13 11:27:46.481463350 +0000 2024-09-13T11:30:33.758 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-13 11:27:46.481463350 +0000 2024-09-13T11:30:33.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:30:33.822 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-13T11:30:33.822 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-13T11:30:33.822 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000199958 s, 2.6 MB/s 2024-09-13T11:30:33.823 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:30:33.880 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:33.880 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:30:33.880 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T11:30:33.899 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T11:30:33.899 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-09-13T11:30:33.957 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T11:30:33.957 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:33.957 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-13T11:30:33.957 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:33.958 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:33.958 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-13 11:30:18.371379549 +0000 2024-09-13T11:30:33.958 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-13 11:27:46.245765269 +0000 2024-09-13T11:30:33.958 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-13 11:27:46.245765269 +0000 2024-09-13T11:30:33.958 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-13 11:27:46.245765269 +0000 2024-09-13T11:30:33.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T11:30:34.024 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-13T11:30:34.024 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-13T11:30:34.024 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000277126 s, 1.8 MB/s 2024-09-13T11:30:34.025 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T11:30:34.084 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-13 11:30:18.371379549 +0000 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-13 11:27:46.573760721 +0000 2024-09-13T11:30:34.141 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-13 11:27:46.573760721 +0000 2024-09-13T11:30:34.142 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-13 11:27:46.573760721 +0000 2024-09-13T11:30:34.142 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T11:30:34.206 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-13T11:30:34.207 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-13T11:30:34.207 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000233481 s, 2.2 MB/s 2024-09-13T11:30:34.208 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T11:30:34.265 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-13 11:30:18.372379535 +0000 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-13 11:27:46.912756020 +0000 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-13 11:27:46.912756020 +0000 2024-09-13T11:30:34.323 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-13 11:27:46.912756020 +0000 2024-09-13T11:30:34.324 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T11:30:34.388 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-13T11:30:34.388 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-13T11:30:34.388 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000191629 s, 2.7 MB/s 2024-09-13T11:30:34.389 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T11:30:34.446 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-09-13 11:30:18.372379535 +0000 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-09-13 11:27:47.233751569 +0000 2024-09-13T11:30:34.502 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-09-13 11:27:47.233751569 +0000 2024-09-13T11:30:34.503 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-09-13 11:27:47.233751569 +0000 2024-09-13T11:30:34.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T11:30:34.566 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-09-13T11:30:34.566 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-09-13T11:30:34.566 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000184548 s, 2.8 MB/s 2024-09-13T11:30:34.568 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T11:30:34.625 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:34.626 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T11:30:34.626 INFO:tasks.ceph:Generating config... 2024-09-13T11:30:34.627 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-13T11:30:34.627 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-13T11:30:34.627 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-13T11:30:34.627 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T11:30:34.627 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] filestore odsync write = True 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] osd backoff on degraded = True 2024-09-13T11:30:34.628 INFO:tasks.ceph:[osd] osd backoff on peering = True 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd max backfills = 2 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd max scrubs = 3 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T11:30:34.629 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-13T11:30:34.630 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-13T11:30:34.630 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-13T11:30:34.630 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-13T11:30:34.630 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-13T11:30:34.630 INFO:tasks.ceph:[osd] osd snap trim sleep = 0.5 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] ms type = async 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2024-09-13T11:30:34.630 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-13T11:30:34.631 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2024-09-13T11:30:34.631 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-13T11:30:34.631 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-13T11:30:34.631 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-13T11:30:34.632 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T11:30:34.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T11:30:34.700 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T11:30:34.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T11:30:34.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:30:34.777 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.38'), ('mon.b', '172.21.15.118'), ('mon.c', '172.21.15.154')] 2024-09-13T11:30:34.777 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.38,172.21.15.118,172.21.15.154', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore odsync write': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 0.5}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-13T11:30:34.778 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:34.778 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:30:34.834 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --add b 172.21.15.118 --add c 172.21.15.154 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:34.923 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:34.923 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid b332d022-c5e6-4682-9d3d-2b6994cd0f20 2024-09-13T11:30:34.923 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-09-13T11:30:34.923 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2024-09-13T11:30:34.923 INFO:teuthology.orchestra.run.smithi038.stdout:fsid b332d022-c5e6-4682-9d3d-2b6994cd0f20 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2024-09-13T11:30:34.922798+0000 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-09-13T11:30:34.922798+0000 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 17 (quincy) 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 3 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:2: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.c 2024-09-13T11:30:34.924 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T11:30:34.925 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T11:30:34.941 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b332d022-c5e6-4682-9d3d-2b6994cd0f20... 2024-09-13T11:30:34.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:30:34.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:30:34.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:30:34.988 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T11:30:35.026 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-09-13T11:30:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2024-09-13T11:30:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:30:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2024-09-13T11:30:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2024-09-13T11:30:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:30:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = "172.21.15.38,172.21.15.118,172.21.15.154" 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: mon client directed command retry = 5 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: mon election default strategy = 3 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: ms bind msgr1 = False 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: ms bind msgr2 = True 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: ms inject socket failures = 5000 2024-09-13T11:30:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: ms type = async 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd_max_pg_log_entries = 2 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd_min_pg_log_entries = 1 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pg_log_trim_min = 0 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_min_size = 2 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_size = 3 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = b332d022-c5e6-4682-9d3d-2b6994cd0f20 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2024-09-13T11:30:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 120 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug trim objects = true 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:30:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: debug monc = 20 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: bdev async discard = True 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: bdev enable discard = True 2024-09-13T11:30:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore block size = 96636764160 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore fsck on mount = True 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore write v2 random = True 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore zero block detection = True 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 20 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 20 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 20 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 10 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: filestore odsync write = True 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd full ratio = 0.9 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:30:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: osd backoff on degraded = True 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd backoff on peering = True 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify cached snaps = True 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify missing on start = True 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd heartbeat use min delay socket = True 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd max backfills = 2 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd max markdown count = 1000 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd max scrubs = 3 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock override recovery settings = True 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd memory target = 1610612736 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd objectstore = bluestore 2024-09-13T11:30:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: osd op queue = debug_random 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: osd op queue cut off = debug_random 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub min interval = 60 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: osd shutdown pgref assert = True 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: osd snap trim sleep = 0.5 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2024-09-13T11:30:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:30:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: # 1m isn't quite enough 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon min osdmap epochs = 50 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune interval = 2 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune min = 15 2024-09-13T11:30:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: mon scrub interval = 300 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: paxos service trim min = 10 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2024-09-13T11:30:35.038 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.b] 2024-09-13T11:30:35.039 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.c] 2024-09-13T11:30:35.039 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-09-13T11:30:35.039 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2024-09-13T11:30:35.039 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.040 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:30:35.041 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2024-09-13T11:30:35.042 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = "172.21.15.38,172.21.15.118,172.21.15.154" 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: mon client directed command retry = 5 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: mon election default strategy = 3 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: ms bind msgr1 = False 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: ms bind msgr2 = True 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject socket failures = 5000 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: ms type = async 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_pg_log_entries = 2 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: osd_min_pg_log_entries = 1 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pg_log_trim_min = 0 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_min_size = 2 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_size = 3 2024-09-13T11:30:35.043 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = b332d022-c5e6-4682-9d3d-2b6994cd0f20 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 120 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2024-09-13T11:30:35.044 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug trim objects = true 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2024-09-13T11:30:35.045 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: bdev async discard = True 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: bdev enable discard = True 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore write v2 random = True 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore zero block detection = True 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 20 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 20 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-13T11:30:35.046 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 10 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: filestore odsync write = True 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd backoff on degraded = True 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd backoff on peering = True 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify cached snaps = True 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify missing on start = True 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd heartbeat use min delay socket = True 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd max backfills = 2 2024-09-13T11:30:35.047 INFO:teuthology.orchestra.run.smithi118.stdout: osd max markdown count = 1000 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd max scrubs = 3 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock override recovery settings = True 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd memory target = 1610612736 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue = debug_random 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue cut off = debug_random 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub min interval = 60 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd shutdown pgref assert = True 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: osd snap trim sleep = 0.5 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-13T11:30:35.048 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2024-09-13T11:30:35.049 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-09-13T11:30:35.049 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-09-13T11:30:35.049 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.049 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.049 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2024-09-13T11:30:35.049 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.049 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:30:35.050 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon min osdmap epochs = 50 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune interval = 2 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune min = 15 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub interval = 300 2024-09-13T11:30:35.051 INFO:teuthology.orchestra.run.smithi118.stdout: paxos service trim min = 10 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.b] 2024-09-13T11:30:35.052 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.c] 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-09-13T11:30:35.053 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-09-13T11:30:35.054 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.055 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = "172.21.15.38,172.21.15.118,172.21.15.154" 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: mon election default strategy = 3 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: ms bind msgr1 = False 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: ms bind msgr2 = True 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: ms type = async 2024-09-13T11:30:35.056 INFO:teuthology.orchestra.run.smithi204.stdout: osd_max_pg_log_entries = 2 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd_min_pg_log_entries = 1 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pg_log_trim_min = 0 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_min_size = 2 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_size = 3 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = b332d022-c5e6-4682-9d3d-2b6994cd0f20 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 120 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:30:35.057 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug trim objects = true 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.058 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: debug monc = 20 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore write v2 random = True 2024-09-13T11:30:35.059 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore zero block detection = True 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 20 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 20 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 10 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: filestore odsync write = True 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: osd backoff on degraded = True 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: osd backoff on peering = True 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:30:35.060 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify cached snaps = True 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify missing on start = True 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd heartbeat use min delay socket = True 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd max backfills = 2 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd max markdown count = 1000 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd max scrubs = 3 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock override recovery settings = True 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd memory target = 1610612736 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue = debug_random 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue cut off = debug_random 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub min interval = 60 2024-09-13T11:30:35.061 INFO:teuthology.orchestra.run.smithi204.stdout: osd shutdown pgref assert = True 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: osd snap trim sleep = 0.5 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-09-13T11:30:35.062 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.063 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon min osdmap epochs = 50 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune interval = 2 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune min = 15 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:30:35.064 INFO:teuthology.orchestra.run.smithi204.stdout: mon scrub interval = 300 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: paxos service trim min = 10 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.b] 2024-09-13T11:30:35.065 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.c] 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2024-09-13T11:30:35.068 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.069 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.070 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = "172.21.15.38,172.21.15.118,172.21.15.154" 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: mon client directed command retry = 5 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: mon election default strategy = 3 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: ms bind msgr1 = False 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: ms bind msgr2 = True 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject socket failures = 5000 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: ms type = async 2024-09-13T11:30:35.071 INFO:teuthology.orchestra.run.smithi154.stdout: osd_max_pg_log_entries = 2 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd_min_pg_log_entries = 1 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pg_log_trim_min = 0 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_min_size = 2 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_size = 3 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = b332d022-c5e6-4682-9d3d-2b6994cd0f20 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 120 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock profile = high_recovery_ops 2024-09-13T11:30:35.072 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug trim objects = true 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2024-09-13T11:30:35.073 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: debug monc = 20 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: bdev async discard = True 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: bdev enable discard = True 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore block size = 96636764160 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore fsck on mount = True 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore write v2 random = True 2024-09-13T11:30:35.074 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore zero block detection = True 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluefs = 20 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluestore = 20 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 20 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: debug rocksdb = 10 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: filestore odsync write = True 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd full ratio = 0.9 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd nearfull ratio = 0.8 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: osd backoff on degraded = True 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: osd backoff on peering = True 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: osd blocked scrub grace period = 3600 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify cached snaps = True 2024-09-13T11:30:35.075 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify missing on start = True 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd failsafe full ratio = 0.95 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd heartbeat use min delay socket = True 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd max backfills = 2 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd max markdown count = 1000 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd max scrubs = 3 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock override recovery settings = True 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd memory target = 1610612736 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd objectstore = bluestore 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue = debug_random 2024-09-13T11:30:35.076 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue cut off = debug_random 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub min interval = 60 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: osd shutdown pgref assert = True 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: osd snap trim sleep = 0.5 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.077 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T11:30:35.078 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: # 1m isn't quite enough 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon_down_mkfs_grace = 2m 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_filestore_osds = false 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon min osdmap epochs = 50 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune interval = 2 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune min = 15 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune txsize = 2 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: mon scrub interval = 300 2024-09-13T11:30:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: paxos service trim min = 10 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.b] 2024-09-13T11:30:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.c] 2024-09-13T11:30:35.080 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T11:30:35.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T11:30:35.163 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T11:30:35.163 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:35.163 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T11:30:35.179 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:35.179 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T11:30:35.234 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:30:35.234 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:35.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:30:35.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:30:35.312 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:30:35.313 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:35.369 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:30:35.369 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:30:35.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:30:35.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:30:35.409 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:30:35.409 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:35.466 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:30:35.466 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:30:35.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:30:35.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:30:35.506 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:30:35.506 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:35.563 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:30:35.563 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:30:35.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T11:30:35.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T11:30:35.600 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:30:35.600 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:35.656 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T11:30:35.656 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T11:30:35.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T11:30:35.709 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T11:30:35.712 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:30:35.783 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T11:30:35.784 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T11:30:35.784 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T11:30:35.784 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T11:30:35.825 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T11:30:35.838 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T11:30:35.839 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-13T11:30:35.839 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T11:30:35.865 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:35.865 INFO:tasks.ceph:role: osd.0 2024-09-13T11:30:35.865 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:30:35.866 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:30:35.959 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:35.960 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:37.281 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-09-13T11:30:37.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-09-13T11:30:37.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T11:30:37.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T11:30:37.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T11:30:37.465 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:37.466 INFO:tasks.ceph:role: osd.4 2024-09-13T11:30:37.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:30:37.466 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:37.563 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:37.564 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:38.884 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-09-13T11:30:38.887 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-09-13T11:30:38.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T11:30:38.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T11:30:38.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-13T11:30:39.064 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:39.064 INFO:tasks.ceph:role: osd.8 2024-09-13T11:30:39.065 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:30:39.065 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:30:39.162 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:39.162 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:39.162 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:39.163 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:39.163 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:40.499 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-09-13T11:30:40.502 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-09-13T11:30:40.503 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-13T11:30:40.540 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-13T11:30:40.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-13T11:30:40.681 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:40.681 INFO:tasks.ceph:role: osd.12 2024-09-13T11:30:40.681 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:30:40.681 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:40.778 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:40.779 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:42.115 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-09-13T11:30:42.117 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-09-13T11:30:42.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-13T11:30:42.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-13T11:30:42.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:42.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:42.331+0000 7f209f422640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T11:30:42.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:42.332+0000 7f209f422640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T11:30:42.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:42.332+0000 7f209f422640 -1 bdev(0x563adad5b800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T11:30:42.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:42.332+0000 7f209f422640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T11:30:45.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T11:30:45.306 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:45.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:45.397+0000 7f9a022a5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T11:30:45.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:45.398+0000 7f9a022a5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T11:30:45.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:45.398+0000 7f9a022a5640 -1 bdev(0x55698c745800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T11:30:45.399 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:45.398+0000 7f9a022a5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T11:30:48.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T11:30:48.602 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:48.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:48.695+0000 7ff934416640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-13T11:30:48.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:48.696+0000 7ff934416640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-13T11:30:48.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:48.696+0000 7ff934416640 -1 bdev(0x5576535c7800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-13T11:30:48.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:48.696+0000 7ff934416640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-13T11:30:51.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-13T11:30:51.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:51.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:51.991+0000 7fd4a03d4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-13T11:30:51.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:51.991+0000 7fd4a03d4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-13T11:30:51.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:51.991+0000 7fd4a03d4640 -1 bdev(0x562399439800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-13T11:30:51.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:30:51.991+0000 7fd4a03d4640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-13T11:30:54.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-13T11:30:54.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T11:30:54.983 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:54.983 INFO:tasks.ceph:role: osd.1 2024-09-13T11:30:54.983 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:30:54.984 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:30:55.052 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:55.053 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:55.085 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-13T11:30:55.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-13T11:30:55.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-13T11:30:55.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T11:30:55.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T11:30:55.297 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:55.298 INFO:tasks.ceph:role: osd.5 2024-09-13T11:30:55.298 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:30:55.298 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:55.362 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:55.394 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-13T11:30:55.397 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-13T11:30:55.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T11:30:55.470 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T11:30:55.539 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-13T11:30:55.604 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:55.604 INFO:tasks.ceph:role: osd.9 2024-09-13T11:30:55.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:30:55.605 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:30:55.669 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:55.669 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:55.669 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:55.669 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:55.669 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:55.670 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:55.670 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:55.670 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:55.670 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:55.670 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:55.703 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-13T11:30:55.704 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-13T11:30:55.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-13T11:30:55.772 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-13T11:30:55.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-13T11:30:55.906 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-13T11:30:55.906 INFO:tasks.ceph:role: osd.13 2024-09-13T11:30:55.906 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:30:55.906 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:30:55.971 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:30:56.003 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-13T11:30:56.005 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-13T11:30:56.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-13T11:30:56.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-13T11:30:56.142 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:56.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:56.241+0000 7f399b42e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T11:30:56.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:56.241+0000 7f399b42e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T11:30:56.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:56.241+0000 7f399b42e640 -1 bdev(0x559557ca9800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T11:30:56.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:56.242+0000 7f399b42e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T11:30:59.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T11:30:59.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:30:59.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:59.769+0000 7fb8f0923640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T11:30:59.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:59.769+0000 7fb8f0923640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T11:30:59.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:59.770+0000 7fb8f0923640 -1 bdev(0x562aaaea1800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T11:30:59.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:30:59.770+0000 7fb8f0923640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T11:31:02.687 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T11:31:02.718 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:02.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:02.810+0000 7f2f9a20d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-13T11:31:02.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:02.810+0000 7f2f9a20d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-13T11:31:02.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:02.810+0000 7f2f9a20d640 -1 bdev(0x55d29b8b3800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:02.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:02.810+0000 7f2f9a20d640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-13T11:31:06.222 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-13T11:31:06.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:06.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:06.346+0000 7f2138cd6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-13T11:31:06.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:06.347+0000 7f2138cd6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-13T11:31:06.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:06.347+0000 7f2138cd6640 -1 bdev(0x560696bed800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:06.349 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-13T11:31:06.347+0000 7f2138cd6640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-13T11:31:09.766 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-13T11:31:09.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T11:31:09.827 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:09.828 INFO:tasks.ceph:role: osd.2 2024-09-13T11:31:09.828 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:31:09.828 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:31:09.903 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:09.903 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:09.903 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:09.904 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:09.904 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:09.904 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:09.904 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:09.904 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:09.904 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:09.904 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:10.402 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-13T11:31:10.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-13T11:31:10.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-13T11:31:10.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T11:31:10.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T11:31:10.612 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:10.612 INFO:tasks.ceph:role: osd.6 2024-09-13T11:31:10.612 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:31:10.613 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:10.685 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:10.686 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:10.686 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:11.141 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-13T11:31:11.143 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-13T11:31:11.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-13T11:31:11.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T11:31:11.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-13T11:31:11.349 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:11.350 INFO:tasks.ceph:role: osd.10 2024-09-13T11:31:11.350 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:31:11.350 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:31:11.421 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:11.421 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:11.422 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:11.906 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-13T11:31:11.908 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-13T11:31:11.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-13T11:31:12.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-13T11:31:12.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-13T11:31:12.101 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:12.101 INFO:tasks.ceph:role: osd.14 2024-09-13T11:31:12.101 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:31:12.102 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:31:12.174 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:12.175 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:12.683 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-13T11:31:12.685 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-13T11:31:12.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-13T11:31:12.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-13T11:31:12.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:12.974 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:12.972+0000 7fd89fa1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T11:31:12.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:12.975+0000 7fd89fa1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T11:31:12.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:12.975+0000 7fd89fa1e640 -1 bdev(0x55be0b415800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:12.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:12.975+0000 7fd89fa1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T11:31:16.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T11:31:16.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:16.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:16.258+0000 7fd21120d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T11:31:16.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:16.258+0000 7fd21120d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T11:31:16.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:16.258+0000 7fd21120d640 -1 bdev(0x55b84098f800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:16.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:16.258+0000 7fd21120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T11:31:19.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T11:31:19.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:19.781 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:19.779+0000 7f45a5ddd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-13T11:31:19.781 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:19.779+0000 7f45a5ddd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-13T11:31:19.782 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:19.780+0000 7f45a5ddd640 -1 bdev(0x557d43f67800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:19.782 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:19.780+0000 7f45a5ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-13T11:31:22.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-13T11:31:22.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:23.058 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:23.056+0000 7f65af42e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-13T11:31:23.058 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:23.056+0000 7f65af42e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-13T11:31:23.058 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:23.056+0000 7f65af42e640 -1 bdev(0x56432c4e5800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:23.059 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-13T11:31:23.056+0000 7f65af42e640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-13T11:31:25.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-13T11:31:26.006 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T11:31:26.037 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:26.037 INFO:tasks.ceph:role: osd.3 2024-09-13T11:31:26.037 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:31:26.037 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T11:31:26.113 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:26.113 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:26.113 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:26.114 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:26.114 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:26.114 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:26.114 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:26.622 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-13T11:31:26.624 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-13T11:31:26.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T11:31:26.779 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T11:31:26.812 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T11:31:26.878 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:26.878 INFO:tasks.ceph:role: osd.7 2024-09-13T11:31:26.878 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:31:26.878 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T11:31:26.952 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:26.953 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:27.393 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-13T11:31:27.395 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-13T11:31:27.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-13T11:31:27.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T11:31:27.546 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-13T11:31:27.611 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:27.611 INFO:tasks.ceph:role: osd.11 2024-09-13T11:31:27.611 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:31:27.611 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:27.683 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:28.170 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-13T11:31:28.172 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-13T11:31:28.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-13T11:31:28.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-13T11:31:28.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-13T11:31:28.417 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-13T11:31:28.417 INFO:tasks.ceph:role: osd.15 2024-09-13T11:31:28.417 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:31:28.417 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T11:31:28.490 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T11:31:28.998 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-09-13T11:31:29.000 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-09-13T11:31:29.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-13T11:31:29.141 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-13T11:31:29.173 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:29.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:29.276+0000 7f8aee0a5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T11:31:29.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:29.276+0000 7f8aee0a5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T11:31:29.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:29.276+0000 7f8aee0a5640 -1 bdev(0x556241e21800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:29.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:29.276+0000 7f8aee0a5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T11:31:32.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T11:31:32.464 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:32.554 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:32.552+0000 7f838a20e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T11:31:32.554 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:32.553+0000 7f838a20e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T11:31:32.554 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:32.553+0000 7f838a20e640 -1 bdev(0x5570de119800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:32.554 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:32.553+0000 7f838a20e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T11:31:35.489 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T11:31:35.518 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:35.612 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:35.610+0000 7f800f420640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-13T11:31:35.612 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:35.610+0000 7f800f420640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-13T11:31:35.612 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:35.610+0000 7f800f420640 -1 bdev(0x55fc3443d800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:35.612 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:35.610+0000 7f800f420640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-13T11:31:39.034 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-13T11:31:39.064 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:39.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:39.154+0000 7febf1c2c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-13T11:31:39.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:39.154+0000 7febf1c2c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-13T11:31:39.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:39.154+0000 7febf1c2c640 -1 bdev(0x5653d61e7800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-13T11:31:39.156 INFO:teuthology.orchestra.run.smithi204.stderr:2024-09-13T11:31:39.154+0000 7febf1c2c640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-13T11:31:42.308 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-13T11:31:42.337 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T11:31:42.338 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:42.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T11:31:42.370 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:42.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T11:31:42.440 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:42.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T11:31:42.509 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:42.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-13T11:31:42.577 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:42.577 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-13T11:31:42.646 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:42.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T11:31:42.676 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:42.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T11:31:42.745 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:42.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-13T11:31:42.815 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:42.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-13T11:31:42.882 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:42.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T11:31:42.913 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:42.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T11:31:42.981 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:42.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-13T11:31:43.048 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:43.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-13T11:31:43.115 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:43.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T11:31:43.145 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:43.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T11:31:43.215 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:43.215 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T11:31:43.284 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:43.284 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-13T11:31:43.352 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:43.352 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-13T11:31:43.419 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:43.420 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T11:31:43.477 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T11:31:43.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:31:43.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:31:43.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T11:31:43.509 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.y] 2024-09-13T11:31:43.509 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBbIuRm+aJJKhAAMxNSj3KXGKP9FerGflivhQ== 2024-09-13T11:31:43.509 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBiIuRmu4rMExAABq2AbjGU/7OYFdOn6+Ik/g== 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.4] 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBlIuRm7yu8FxAAuhDRU94Eu+or3ogBznjF/g== 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.8] 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBoIuRmpVCDKRAAaB6zCIdcGxkNbVXKwvJo1Q== 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.12] 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBrIuRm5DgeOxAAjxXpflumFln341kL/YnxJw== 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBwIuRmGyZ4DhAA58PctxRNNVvesl8m5NInuQ== 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.5] 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBzIuRmOMXxLRAAiQeY6mstL+NaK2EaPueU0w== 2024-09-13T11:31:43.510 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.9] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQB2IuRmp7BbMBAAMlAskPGQNG8IgRw+942CSg== 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.13] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQB6IuRmY327FBAAPSFF83/jOeqfDIIJ1znJ/Q== 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCAIuRmis4MOhAA9m0VsZOMC7Q3hVd5RCN+RQ== 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.6] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCEIuRmoQF5DxAAls5wS/F3inY32LMkr1NxQw== 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.10] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCHIuRmRV6JLhAAxha3OOB9qdaBBZrwnvSbqw== 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.14] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCLIuRmtZpoAxAASDm1U+R/uj1Wji03KexKng== 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBbIuRmwfCSLhAAroqVL+88LtZ4ZfZ1Ovk2eg== 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.3] 2024-09-13T11:31:43.511 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCRIuRmQBuDEBAA0qNQs+IC1Li5i4qwfg3ONw== 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.7] 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCUIuRmyp8AIRAA6yy4jyUUQktgGA5AziczDg== 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.11] 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCXIuRmxHZvJBAAv0BTiPnxVS0YkzGisGOLsw== 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.15] 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQCbIuRmRME+CRAAc76VgVODg3OTOrgghER1GQ== 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-09-13T11:31:43.512 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQBbIuRmNiMzMRAAfCwtDmRhvFLWaiWMUrhfWA== 2024-09-13T11:31:43.514 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.y] 2024-09-13T11:31:43.514 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBbIuRm+aJJKhAAMxNSj3KXGKP9FerGflivhQ== 2024-09-13T11:31:43.514 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-09-13T11:31:43.514 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBiIuRmu4rMExAABq2AbjGU/7OYFdOn6+Ik/g== 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.4] 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBlIuRm7yu8FxAAuhDRU94Eu+or3ogBznjF/g== 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.8] 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBoIuRmpVCDKRAAaB6zCIdcGxkNbVXKwvJo1Q== 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.12] 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBrIuRm5DgeOxAAjxXpflumFln341kL/YnxJw== 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBwIuRmGyZ4DhAA58PctxRNNVvesl8m5NInuQ== 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.5] 2024-09-13T11:31:43.515 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBzIuRmOMXxLRAAiQeY6mstL+NaK2EaPueU0w== 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.9] 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQB2IuRmp7BbMBAAMlAskPGQNG8IgRw+942CSg== 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.13] 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQB6IuRmY327FBAAPSFF83/jOeqfDIIJ1znJ/Q== 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCAIuRmis4MOhAA9m0VsZOMC7Q3hVd5RCN+RQ== 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.6] 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCEIuRmoQF5DxAAls5wS/F3inY32LMkr1NxQw== 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.10] 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCHIuRmRV6JLhAAxha3OOB9qdaBBZrwnvSbqw== 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.14] 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCLIuRmtZpoAxAASDm1U+R/uj1Wji03KexKng== 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2024-09-13T11:31:43.516 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBbIuRmwfCSLhAAroqVL+88LtZ4ZfZ1Ovk2eg== 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCRIuRmQBuDEBAA0qNQs+IC1Li5i4qwfg3ONw== 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.7] 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCUIuRmyp8AIRAA6yy4jyUUQktgGA5AziczDg== 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.11] 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCXIuRmxHZvJBAAv0BTiPnxVS0YkzGisGOLsw== 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.15] 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQCbIuRmRME+CRAAc76VgVODg3OTOrgghER1GQ== 2024-09-13T11:31:43.517 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBbIuRmNiMzMRAAfCwtDmRhvFLWaiWMUrhfWA== 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.y] 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBbIuRm+aJJKhAAMxNSj3KXGKP9FerGflivhQ== 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBiIuRmu4rMExAABq2AbjGU/7OYFdOn6+Ik/g== 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.4] 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBlIuRm7yu8FxAAuhDRU94Eu+or3ogBznjF/g== 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.8] 2024-09-13T11:31:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBoIuRmpVCDKRAAaB6zCIdcGxkNbVXKwvJo1Q== 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.12] 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBrIuRm5DgeOxAAjxXpflumFln341kL/YnxJw== 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBwIuRmGyZ4DhAA58PctxRNNVvesl8m5NInuQ== 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.5] 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBzIuRmOMXxLRAAiQeY6mstL+NaK2EaPueU0w== 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.9] 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQB2IuRmp7BbMBAAMlAskPGQNG8IgRw+942CSg== 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.13] 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQB6IuRmY327FBAAPSFF83/jOeqfDIIJ1znJ/Q== 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCAIuRmis4MOhAA9m0VsZOMC7Q3hVd5RCN+RQ== 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.6] 2024-09-13T11:31:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCEIuRmoQF5DxAAls5wS/F3inY32LMkr1NxQw== 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.10] 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCHIuRmRV6JLhAAxha3OOB9qdaBBZrwnvSbqw== 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.14] 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCLIuRmtZpoAxAASDm1U+R/uj1Wji03KexKng== 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBbIuRmwfCSLhAAroqVL+88LtZ4ZfZ1Ovk2eg== 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.3] 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCRIuRmQBuDEBAA0qNQs+IC1Li5i4qwfg3ONw== 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.7] 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCUIuRmyp8AIRAA6yy4jyUUQktgGA5AziczDg== 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.11] 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCXIuRmxHZvJBAAv0BTiPnxVS0YkzGisGOLsw== 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.15] 2024-09-13T11:31:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCbIuRmRME+CRAAc76VgVODg3OTOrgghER1GQ== 2024-09-13T11:31:43.521 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2024-09-13T11:31:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQBbIuRmNiMzMRAAfCwtDmRhvFLWaiWMUrhfWA== 2024-09-13T11:31:43.521 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:31:43.557 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:31:43.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:31:43.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.620 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.678 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.739 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.912 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:43.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.132 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.135 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:31:44.302 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:31:44.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T11:31:44.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.418 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T11:31:44.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:31:44.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:31:44.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T11:31:44.643 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T11:31:44.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T11:31:44.670 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:31:44.788 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T11:31:44.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T11:31:44.840 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:31:44.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T11:31:44.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T11:31:44.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T11:31:45.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T11:31:45.108 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:45.111 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:45.116 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T11:31:45.164 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T11:31:45.164 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T11:31:45.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T11:31:45.167 INFO:tasks.ceph.mon.a:Started 2024-09-13T11:31:45.167 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T11:31:45.167 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T11:31:45.169 INFO:tasks.ceph.mon.b:Started 2024-09-13T11:31:45.169 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T11:31:45.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T11:31:45.207 INFO:tasks.ceph.mon.c:Started 2024-09-13T11:31:45.207 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T11:31:45.207 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T11:31:45.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T11:31:45.209 INFO:tasks.ceph.mgr.y:Started 2024-09-13T11:31:45.210 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T11:31:45.210 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T11:31:45.212 INFO:tasks.ceph.mgr.x:Started 2024-09-13T11:31:45.212 DEBUG:tasks.ceph:set 0 configs 2024-09-13T11:31:45.213 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T11:31:45.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T11:31:50.603 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-09-13T11:31:50.619 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T11:31:50.620 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T11:31:50.620 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:50.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T11:31:50.647 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:50.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T11:31:50.713 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:50.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-13T11:31:50.780 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-09-13T11:31:50.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-13T11:31:50.846 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:50.847 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T11:31:50.877 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:50.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T11:31:50.945 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:50.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-13T11:31:51.013 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-13T11:31:51.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-13T11:31:51.079 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:51.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T11:31:51.109 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:51.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T11:31:51.178 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:51.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-13T11:31:51.246 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-13T11:31:51.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-13T11:31:51.313 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:51.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T11:31:51.338 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:51.338 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T11:31:51.403 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:51.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-13T11:31:51.468 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-09-13T11:31:51.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-13T11:31:51.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 5a4d0d43-6562-441e-925e-4c992ee7b335 0 2024-09-13T11:31:51.752 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-09-13T11:31:51.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 80665caa-6933-4f89-b454-128057e9ad92 1 2024-09-13T11:31:51.945 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-09-13T11:31:51.956 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new eb99fd51-a064-4311-932e-8b6d555409af 2 2024-09-13T11:31:52.138 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-09-13T11:31:52.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 3d840ed9-a410-4bcb-b0f0-70fafdf69842 3 2024-09-13T11:31:52.333 INFO:teuthology.orchestra.run.smithi204.stdout:3 2024-09-13T11:31:52.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 0c01b7d7-1309-40fa-913a-97d2a8499293 4 2024-09-13T11:31:52.524 INFO:teuthology.orchestra.run.smithi204.stdout:4 2024-09-13T11:31:52.535 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new b4cec928-1764-4fe0-8c5d-870d8a0c15c7 5 2024-09-13T11:31:52.715 INFO:teuthology.orchestra.run.smithi204.stdout:5 2024-09-13T11:31:52.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 89f6d810-2afc-42b0-b989-052218b6887c 6 2024-09-13T11:31:52.905 INFO:teuthology.orchestra.run.smithi204.stdout:6 2024-09-13T11:31:52.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new ccfee49f-f394-4de0-8e30-853f38e31232 7 2024-09-13T11:31:53.145 INFO:teuthology.orchestra.run.smithi204.stdout:7 2024-09-13T11:31:53.156 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 03b1c505-1b92-4daa-9da4-c254be2ba792 8 2024-09-13T11:31:53.336 INFO:teuthology.orchestra.run.smithi204.stdout:8 2024-09-13T11:31:53.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 8fbee7a0-2ee6-4551-bf06-4e013e413fb4 9 2024-09-13T11:31:53.525 INFO:teuthology.orchestra.run.smithi204.stdout:9 2024-09-13T11:31:53.536 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 0808cb41-f62e-49b0-935d-616bea1a4db5 10 2024-09-13T11:31:53.715 INFO:teuthology.orchestra.run.smithi204.stdout:10 2024-09-13T11:31:53.726 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 427dd574-f2a8-49e5-a32a-cfda023a514a 11 2024-09-13T11:31:53.907 INFO:teuthology.orchestra.run.smithi204.stdout:11 2024-09-13T11:31:53.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 17942689-1130-4b6b-9f98-1605a03731b1 12 2024-09-13T11:31:54.097 INFO:teuthology.orchestra.run.smithi204.stdout:12 2024-09-13T11:31:54.109 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 62161cab-6d14-4571-971d-217601e38fbc 13 2024-09-13T11:31:54.287 INFO:teuthology.orchestra.run.smithi204.stdout:13 2024-09-13T11:31:54.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 559ef141-27db-452d-9577-a6135421f64f 14 2024-09-13T11:31:54.477 INFO:teuthology.orchestra.run.smithi204.stdout:14 2024-09-13T11:31:54.488 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 98f9ceba-79df-410a-9788-393057e79c30 15 2024-09-13T11:31:54.666 INFO:teuthology.orchestra.run.smithi204.stdout:15 2024-09-13T11:31:54.677 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T11:31:54.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T11:31:54.680 INFO:tasks.ceph.osd.0:Started 2024-09-13T11:31:54.680 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T11:31:54.680 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T11:31:54.682 INFO:tasks.ceph.osd.4:Started 2024-09-13T11:31:54.682 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-13T11:31:54.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-13T11:31:54.685 INFO:tasks.ceph.osd.8:Started 2024-09-13T11:31:54.685 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-13T11:31:54.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-13T11:31:54.690 INFO:tasks.ceph.osd.12:Started 2024-09-13T11:31:54.690 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T11:31:54.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T11:31:54.693 INFO:tasks.ceph.osd.1:Started 2024-09-13T11:31:54.693 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T11:31:54.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T11:31:54.696 INFO:tasks.ceph.osd.5:Started 2024-09-13T11:31:54.696 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-13T11:31:54.696 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-13T11:31:54.698 INFO:tasks.ceph.osd.9:Started 2024-09-13T11:31:54.699 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-13T11:31:54.699 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-13T11:31:54.702 INFO:tasks.ceph.osd.13:Started 2024-09-13T11:31:54.702 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T11:31:54.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T11:31:54.704 INFO:tasks.ceph.osd.2:Started 2024-09-13T11:31:54.705 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T11:31:54.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T11:31:54.707 INFO:tasks.ceph.osd.6:Started 2024-09-13T11:31:54.707 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-13T11:31:54.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-13T11:31:54.709 INFO:tasks.ceph.osd.10:Started 2024-09-13T11:31:54.710 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-13T11:31:54.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-13T11:31:54.712 INFO:tasks.ceph.osd.14:Started 2024-09-13T11:31:54.713 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T11:31:54.713 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T11:31:54.715 INFO:tasks.ceph.osd.3:Started 2024-09-13T11:31:54.715 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T11:31:54.715 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T11:31:54.717 INFO:tasks.ceph.osd.7:Started 2024-09-13T11:31:54.717 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-13T11:31:54.717 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-13T11:31:54.719 INFO:tasks.ceph.osd.11:Started 2024-09-13T11:31:54.720 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-13T11:31:54.720 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-13T11:31:54.722 INFO:tasks.ceph.osd.15:Started 2024-09-13T11:31:54.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:31:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:31:54.921 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":18,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:31:54.664226+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:31:54.664226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:31:54.935 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T11:31:54.935 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T11:31:55.625 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:31:55.623+0000 7f8f12220640 -1 Falling back to public interface 2024-09-13T11:31:55.651 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:31:55.649+0000 7f8a90610640 -1 Falling back to public interface 2024-09-13T11:31:55.677 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:31:55.676+0000 7f0a83e17640 -1 Falling back to public interface 2024-09-13T11:31:55.693 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:31:55.691+0000 7f955f52c640 -1 Falling back to public interface 2024-09-13T11:31:55.886 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:31:55.884+0000 7f3f83ab6640 -1 Falling back to public interface 2024-09-13T11:31:55.896 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:31:55.894+0000 7f4eb542c640 -1 Falling back to public interface 2024-09-13T11:31:55.902 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:31:55.900+0000 7f44022e6640 -1 Falling back to public interface 2024-09-13T11:31:55.917 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:31:55.915+0000 7ff420811640 -1 Falling back to public interface 2024-09-13T11:31:55.929 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:31:55.927+0000 7f7a7b610640 -1 Falling back to public interface 2024-09-13T11:31:55.935 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-09-13T11:31:55.933+0000 7fc0cc4e7640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T11:31:55.935 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-09-13T11:31:55.933+0000 7fc0cc4e7640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T11:31:56.142 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:31:56.141+0000 7fc30e00b640 -1 Falling back to public interface 2024-09-13T11:31:56.143 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:31:56.141+0000 7f36aaa0e640 -1 Falling back to public interface 2024-09-13T11:31:56.174 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:31:56.172+0000 7fd895745640 -1 Falling back to public interface 2024-09-13T11:31:56.222 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:31:56.220+0000 7fc293210640 -1 Falling back to public interface 2024-09-13T11:31:56.376 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:31:56.374+0000 7fb9bbbdb640 -1 Falling back to public interface 2024-09-13T11:31:56.389 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:31:56.387+0000 7fbc03228640 -1 Falling back to public interface 2024-09-13T11:31:56.432 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:31:56.430+0000 7fb48d019640 -1 Falling back to public interface 2024-09-13T11:31:56.544 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:31:56.847 INFO:teuthology.misc.health.smithi038.stdout: 2024-09-13T11:31:56.847 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":18,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:31:54.664226+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T11:31:54.664226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:31:56.859 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-13T11:31:59.223 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:31:59.222+0000 7f0a83e17640 -1 osd.2 0 log_to_monitors true 2024-09-13T11:31:59.271 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:31:59.269+0000 7f955f52c640 -1 osd.11 0 log_to_monitors true 2024-09-13T11:31:59.426 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:31:59.424+0000 7f8f12220640 -1 osd.4 0 log_to_monitors true 2024-09-13T11:31:59.694 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:31:59.692+0000 7f3f83ab6640 -1 osd.9 0 log_to_monitors true 2024-09-13T11:31:59.718 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:31:59.716+0000 7f44022e6640 -1 osd.0 0 log_to_monitors true 2024-09-13T11:31:59.746 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:31:59.744+0000 7fc293210640 -1 osd.7 0 log_to_monitors true 2024-09-13T11:31:59.894 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:31:59.892+0000 7f36aaa0e640 -1 osd.8 0 log_to_monitors true 2024-09-13T11:31:59.920 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:31:59.918+0000 7f8a90610640 -1 osd.5 0 log_to_monitors true 2024-09-13T11:31:59.931 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:31:59.929+0000 7f4eb542c640 -1 osd.1 0 log_to_monitors true 2024-09-13T11:31:59.946 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:31:59.945+0000 7f7a7b610640 -1 osd.14 0 log_to_monitors true 2024-09-13T11:32:00.002 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:00.000+0000 7ff420811640 -1 osd.12 0 log_to_monitors true 2024-09-13T11:32:00.139 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:00.138+0000 7fbc03228640 -1 osd.10 0 log_to_monitors true 2024-09-13T11:32:00.158 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:00.156+0000 7fc30e00b640 -1 osd.6 0 log_to_monitors true 2024-09-13T11:32:00.169 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:00.167+0000 7fb48d019640 -1 osd.15 0 log_to_monitors true 2024-09-13T11:32:00.219 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:00.217+0000 7fd895745640 -1 osd.3 0 log_to_monitors true 2024-09-13T11:32:00.286 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:00.283+0000 7f955b4bb640 -1 osd.11 0 waiting for initial osdmap 2024-09-13T11:32:00.286 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:00.284+0000 7f0a7f801640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T11:32:00.289 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:00.287+0000 7f95572e2640 -1 osd.11 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:00.289 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:00.288+0000 7f0a7b628640 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:00.354 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:00.352+0000 7fb9bbbdb640 -1 osd.13 0 log_to_monitors true 2024-09-13T11:32:01.294 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:01.293+0000 7f8a8bfd9640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T11:32:01.295 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:01.292+0000 7fd8916d4640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T11:32:01.297 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:01.295+0000 7f36a6286640 -1 osd.8 0 waiting for initial osdmap 2024-09-13T11:32:01.300 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:01.298+0000 7ff41c11e640 -1 osd.12 0 waiting for initial osdmap 2024-09-13T11:32:01.300 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:01.299+0000 7f7a76fd9640 -1 osd.14 0 waiting for initial osdmap 2024-09-13T11:32:01.301 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:01.299+0000 7fbbff1b7640 -1 osd.10 0 waiting for initial osdmap 2024-09-13T11:32:01.301 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:01.299+0000 7fc309a4b640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T11:32:01.301 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:01.300+0000 7f8f0db11640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T11:32:01.305 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:01.304+0000 7fbbfafde640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:01.305 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:01.304+0000 7f7a72e00640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:01.306 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:01.305+0000 7fc305872640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:01.310 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:01.308+0000 7f36a20ad640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:01.311 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:01.309+0000 7f8f09938640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:01.312 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:01.309+0000 7ff417f45640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:01.324 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:01.322+0000 7f8a87e00640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:01.463 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:01.460+0000 7fd88d4fb640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:02.027 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:02.025+0000 7f43fe275640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T11:32:02.033 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:02.031+0000 7f43f6094640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:02.075 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:02.073+0000 7f3f7fa45640 -1 osd.9 0 waiting for initial osdmap 2024-09-13T11:32:02.081 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:02.079+0000 7f3f77864640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:02.088 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:02.086+0000 7f4eb124e640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T11:32:02.092 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:02.091+0000 7f4ea8859640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:02.368 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:02.365+0000 7fc28ea97640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T11:32:02.374 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:02.371+0000 7fc2868b6640 -1 osd.7 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:02.425 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:02.422+0000 7fb488926640 -1 osd.15 0 waiting for initial osdmap 2024-09-13T11:32:02.429 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:02.427+0000 7fb480745640 -1 osd.15 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:02.814 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:02.812+0000 7fb9b7b6a640 -1 osd.13 0 waiting for initial osdmap 2024-09-13T11:32:02.819 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:02.817+0000 7fb9af989640 -1 osd.13 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:32:04.473 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T11:32:04.793 INFO:teuthology.misc.health.smithi038.stdout: 2024-09-13T11:32:04.793 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":24,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:32:04.317807+0000","last_up_change":"2024-09-13T11:32:03.312547+0000","last_in_change":"2024-09-13T11:31:54.664226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:32:02.947125+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2677506544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2677506544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2677506544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":2677506544}]},"public_addr":"172.21.15.38:6805/2677506544","cluster_addr":"172.21.15.38:6806/2677506544","heartbeat_back_addr":"172.21.15.38:6808/2677506544","heartbeat_front_addr":"172.21.15.38:6807/2677506544","state":["exists","up"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":3561806747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":3561806747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":3561806747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":3561806747}]},"public_addr":"172.21.15.118:6808/3561806747","cluster_addr":"172.21.15.118:6809/3561806747","heartbeat_back_addr":"172.21.15.118:6811/3561806747","heartbeat_front_addr":"172.21.15.118:6810/3561806747","state":["exists","up"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4217282489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4217282489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4217282489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4217282489}]},"public_addr":"172.21.15.154:6800/4217282489","cluster_addr":"172.21.15.154:6801/4217282489","heartbeat_back_addr":"172.21.15.154:6803/4217282489","heartbeat_front_addr":"172.21.15.154:6802/4217282489","state":["exists","up"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2326505959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":2326505959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":2326505959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2326505959}]},"public_addr":"172.21.15.204:6804/2326505959","cluster_addr":"172.21.15.204:6805/2326505959","heartbeat_back_addr":"172.21.15.204:6807/2326505959","heartbeat_front_addr":"172.21.15.204:6806/2326505959","state":["exists","up"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":3107658120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3107658120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3107658120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":3107658120}]},"public_addr":"172.21.15.38:6801/3107658120","cluster_addr":"172.21.15.38:6802/3107658120","heartbeat_back_addr":"172.21.15.38:6804/3107658120","heartbeat_front_addr":"172.21.15.38:6803/3107658120","state":["exists","up"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3800782738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":3800782738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":3800782738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":3800782738}]},"public_addr":"172.21.15.118:6800/3800782738","cluster_addr":"172.21.15.118:6801/3800782738","heartbeat_back_addr":"172.21.15.118:6803/3800782738","heartbeat_front_addr":"172.21.15.118:6802/3800782738","state":["exists","up"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3986362716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3986362716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3986362716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3986362716}]},"public_addr":"172.21.15.154:6808/3986362716","cluster_addr":"172.21.15.154:6809/3986362716","heartbeat_back_addr":"172.21.15.154:6811/3986362716","heartbeat_front_addr":"172.21.15.154:6810/3986362716","state":["exists","up"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1978982202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":1978982202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":1978982202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1978982202}]},"public_addr":"172.21.15.204:6808/1978982202","cluster_addr":"172.21.15.204:6809/1978982202","heartbeat_back_addr":"172.21.15.204:6811/1978982202","heartbeat_front_addr":"172.21.15.204:6810/1978982202","state":["exists","up"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3608171775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3608171775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3608171775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3608171775}]},"public_addr":"172.21.15.38:6813/3608171775","cluster_addr":"172.21.15.38:6814/3608171775","heartbeat_back_addr":"172.21.15.38:6816/3608171775","heartbeat_front_addr":"172.21.15.38:6815/3608171775","state":["exists","up"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2580659844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2580659844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":2580659844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2580659844}]},"public_addr":"172.21.15.118:6804/2580659844","cluster_addr":"172.21.15.118:6805/2580659844","heartbeat_back_addr":"172.21.15.118:6807/2580659844","heartbeat_front_addr":"172.21.15.118:6806/2580659844","state":["exists","up"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2352891695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":2352891695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":2352891695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2352891695}]},"public_addr":"172.21.15.154:6812/2352891695","cluster_addr":"172.21.15.154:6813/2352891695","heartbeat_back_addr":"172.21.15.154:6815/2352891695","heartbeat_front_addr":"172.21.15.154:6814/2352891695","state":["exists","up"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2304836889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":2304836889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":2304836889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2304836889}]},"public_addr":"172.21.15.204:6800/2304836889","cluster_addr":"172.21.15.204:6801/2304836889","heartbeat_back_addr":"172.21.15.204:6803/2304836889","heartbeat_front_addr":"172.21.15.204:6802/2304836889","state":["exists","up"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":1688419254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1688419254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1688419254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":1688419254}]},"public_addr":"172.21.15.38:6809/1688419254","cluster_addr":"172.21.15.38:6810/1688419254","heartbeat_back_addr":"172.21.15.38:6812/1688419254","heartbeat_front_addr":"172.21.15.38:6811/1688419254","state":["exists","up"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":3727425991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":3727425991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":3727425991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":3727425991}]},"public_addr":"172.21.15.118:6812/3727425991","cluster_addr":"172.21.15.118:6813/3727425991","heartbeat_back_addr":"172.21.15.118:6815/3727425991","heartbeat_front_addr":"172.21.15.118:6814/3727425991","state":["exists","up"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":538351706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":538351706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":538351706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":538351706}]},"public_addr":"172.21.15.154:6804/538351706","cluster_addr":"172.21.15.154:6805/538351706","heartbeat_back_addr":"172.21.15.154:6807/538351706","heartbeat_front_addr":"172.21.15.154:6806/538351706","state":["exists","up"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":501019151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":501019151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":501019151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":501019151}]},"public_addr":"172.21.15.204:6812/501019151","cluster_addr":"172.21.15.204:6813/501019151","heartbeat_back_addr":"172.21.15.204:6815/501019151","heartbeat_front_addr":"172.21.15.204:6814/501019151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.680732+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.915157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.198442+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.248734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.448334+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.873463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.149077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.739636+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.850336+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.655711+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.092457+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.246352+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.027609+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.369804+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.979821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.174959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:32:04.805 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-13T11:32:04.805 INFO:tasks.ceph:Creating RBD pool 2024-09-13T11:32:04.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'rbd' created 2024-09-13T11:32:05.348 DEBUG:teuthology.orchestra.run.smithi038:> rbd --cluster ceph pool init rbd 2024-09-13T11:32:08.387 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T11:32:08.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T11:32:08.388 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T11:32:08.702 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:08.714 INFO:teuthology.orchestra.run.smithi038.stdout:[{"version":7,"timestamp":"2024-09-13T11:32:02.814534+0000","name":"","changes":[{"name":"osd.13/osd_mclock_max_capacity_iops_ssd","new_value":"11227.310069"}]}] 2024-09-13T11:32:08.715 INFO:tasks.ceph_manager:config epoch is 7 2024-09-13T11:32:08.715 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T11:32:08.715 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T11:32:08.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T11:32:09.067 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:09.077 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"flags":0,"active_gid":4111,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":622381731}]},"active_addr":"172.21.15.38:6800/622381731","active_change":"2024-09-13T11:31:54.920426+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2025430352}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2959334534}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":772187553}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2316275772}]}]} 2024-09-13T11:32:09.084 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T11:32:09.085 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T11:32:09.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:32:09.384 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:09.384 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:32:08.355322+0000","last_up_change":"2024-09-13T11:32:03.312547+0000","last_in_change":"2024-09-13T11:31:54.664226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:32:02.947125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:32:05.120118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2677506544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2677506544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2677506544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":2677506544}]},"public_addr":"172.21.15.38:6805/2677506544","cluster_addr":"172.21.15.38:6806/2677506544","heartbeat_back_addr":"172.21.15.38:6808/2677506544","heartbeat_front_addr":"172.21.15.38:6807/2677506544","state":["exists","up"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":3561806747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":3561806747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":3561806747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":3561806747}]},"public_addr":"172.21.15.118:6808/3561806747","cluster_addr":"172.21.15.118:6809/3561806747","heartbeat_back_addr":"172.21.15.118:6811/3561806747","heartbeat_front_addr":"172.21.15.118:6810/3561806747","state":["exists","up"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4217282489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4217282489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4217282489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4217282489}]},"public_addr":"172.21.15.154:6800/4217282489","cluster_addr":"172.21.15.154:6801/4217282489","heartbeat_back_addr":"172.21.15.154:6803/4217282489","heartbeat_front_addr":"172.21.15.154:6802/4217282489","state":["exists","up"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2326505959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":2326505959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":2326505959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2326505959}]},"public_addr":"172.21.15.204:6804/2326505959","cluster_addr":"172.21.15.204:6805/2326505959","heartbeat_back_addr":"172.21.15.204:6807/2326505959","heartbeat_front_addr":"172.21.15.204:6806/2326505959","state":["exists","up"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":3107658120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3107658120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3107658120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":3107658120}]},"public_addr":"172.21.15.38:6801/3107658120","cluster_addr":"172.21.15.38:6802/3107658120","heartbeat_back_addr":"172.21.15.38:6804/3107658120","heartbeat_front_addr":"172.21.15.38:6803/3107658120","state":["exists","up"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3800782738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":3800782738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":3800782738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":3800782738}]},"public_addr":"172.21.15.118:6800/3800782738","cluster_addr":"172.21.15.118:6801/3800782738","heartbeat_back_addr":"172.21.15.118:6803/3800782738","heartbeat_front_addr":"172.21.15.118:6802/3800782738","state":["exists","up"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3986362716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3986362716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3986362716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3986362716}]},"public_addr":"172.21.15.154:6808/3986362716","cluster_addr":"172.21.15.154:6809/3986362716","heartbeat_back_addr":"172.21.15.154:6811/3986362716","heartbeat_front_addr":"172.21.15.154:6810/3986362716","state":["exists","up"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1978982202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":1978982202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":1978982202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1978982202}]},"public_addr":"172.21.15.204:6808/1978982202","cluster_addr":"172.21.15.204:6809/1978982202","heartbeat_back_addr":"172.21.15.204:6811/1978982202","heartbeat_front_addr":"172.21.15.204:6810/1978982202","state":["exists","up"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3608171775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3608171775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3608171775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3608171775}]},"public_addr":"172.21.15.38:6813/3608171775","cluster_addr":"172.21.15.38:6814/3608171775","heartbeat_back_addr":"172.21.15.38:6816/3608171775","heartbeat_front_addr":"172.21.15.38:6815/3608171775","state":["exists","up"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2580659844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2580659844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":2580659844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2580659844}]},"public_addr":"172.21.15.118:6804/2580659844","cluster_addr":"172.21.15.118:6805/2580659844","heartbeat_back_addr":"172.21.15.118:6807/2580659844","heartbeat_front_addr":"172.21.15.118:6806/2580659844","state":["exists","up"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2352891695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":2352891695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":2352891695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2352891695}]},"public_addr":"172.21.15.154:6812/2352891695","cluster_addr":"172.21.15.154:6813/2352891695","heartbeat_back_addr":"172.21.15.154:6815/2352891695","heartbeat_front_addr":"172.21.15.154:6814/2352891695","state":["exists","up"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2304836889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":2304836889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":2304836889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2304836889}]},"public_addr":"172.21.15.204:6800/2304836889","cluster_addr":"172.21.15.204:6801/2304836889","heartbeat_back_addr":"172.21.15.204:6803/2304836889","heartbeat_front_addr":"172.21.15.204:6802/2304836889","state":["exists","up"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":1688419254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1688419254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1688419254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":1688419254}]},"public_addr":"172.21.15.38:6809/1688419254","cluster_addr":"172.21.15.38:6810/1688419254","heartbeat_back_addr":"172.21.15.38:6812/1688419254","heartbeat_front_addr":"172.21.15.38:6811/1688419254","state":["exists","up"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":3727425991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":3727425991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":3727425991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":3727425991}]},"public_addr":"172.21.15.118:6812/3727425991","cluster_addr":"172.21.15.118:6813/3727425991","heartbeat_back_addr":"172.21.15.118:6815/3727425991","heartbeat_front_addr":"172.21.15.118:6814/3727425991","state":["exists","up"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":538351706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":538351706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":538351706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":538351706}]},"public_addr":"172.21.15.154:6804/538351706","cluster_addr":"172.21.15.154:6805/538351706","heartbeat_back_addr":"172.21.15.154:6807/538351706","heartbeat_front_addr":"172.21.15.154:6806/538351706","state":["exists","up"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":501019151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":501019151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":501019151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":501019151}]},"public_addr":"172.21.15.204:6812/501019151","cluster_addr":"172.21.15.204:6813/501019151","heartbeat_back_addr":"172.21.15.204:6815/501019151","heartbeat_front_addr":"172.21.15.204:6814/501019151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.680732+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.915157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.198442+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.248734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.448334+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.873463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.149077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.739636+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.850336+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.655711+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.092457+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.246352+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.027609+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.369804+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.979821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.174959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:32:09.397 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T11:32:09.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:32:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:09.697 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":28,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:32:08.355322+0000","last_up_change":"2024-09-13T11:32:03.312547+0000","last_in_change":"2024-09-13T11:31:54.664226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:32:02.947125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:32:05.120118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2677506544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2677506544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2677506544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":2677506544}]},"public_addr":"172.21.15.38:6805/2677506544","cluster_addr":"172.21.15.38:6806/2677506544","heartbeat_back_addr":"172.21.15.38:6808/2677506544","heartbeat_front_addr":"172.21.15.38:6807/2677506544","state":["exists","up"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":3561806747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":3561806747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":3561806747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":3561806747}]},"public_addr":"172.21.15.118:6808/3561806747","cluster_addr":"172.21.15.118:6809/3561806747","heartbeat_back_addr":"172.21.15.118:6811/3561806747","heartbeat_front_addr":"172.21.15.118:6810/3561806747","state":["exists","up"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4217282489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4217282489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4217282489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4217282489}]},"public_addr":"172.21.15.154:6800/4217282489","cluster_addr":"172.21.15.154:6801/4217282489","heartbeat_back_addr":"172.21.15.154:6803/4217282489","heartbeat_front_addr":"172.21.15.154:6802/4217282489","state":["exists","up"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2326505959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":2326505959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":2326505959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2326505959}]},"public_addr":"172.21.15.204:6804/2326505959","cluster_addr":"172.21.15.204:6805/2326505959","heartbeat_back_addr":"172.21.15.204:6807/2326505959","heartbeat_front_addr":"172.21.15.204:6806/2326505959","state":["exists","up"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":3107658120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3107658120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3107658120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":3107658120}]},"public_addr":"172.21.15.38:6801/3107658120","cluster_addr":"172.21.15.38:6802/3107658120","heartbeat_back_addr":"172.21.15.38:6804/3107658120","heartbeat_front_addr":"172.21.15.38:6803/3107658120","state":["exists","up"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3800782738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":3800782738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":3800782738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":3800782738}]},"public_addr":"172.21.15.118:6800/3800782738","cluster_addr":"172.21.15.118:6801/3800782738","heartbeat_back_addr":"172.21.15.118:6803/3800782738","heartbeat_front_addr":"172.21.15.118:6802/3800782738","state":["exists","up"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3986362716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3986362716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3986362716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3986362716}]},"public_addr":"172.21.15.154:6808/3986362716","cluster_addr":"172.21.15.154:6809/3986362716","heartbeat_back_addr":"172.21.15.154:6811/3986362716","heartbeat_front_addr":"172.21.15.154:6810/3986362716","state":["exists","up"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1978982202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":1978982202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":1978982202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1978982202}]},"public_addr":"172.21.15.204:6808/1978982202","cluster_addr":"172.21.15.204:6809/1978982202","heartbeat_back_addr":"172.21.15.204:6811/1978982202","heartbeat_front_addr":"172.21.15.204:6810/1978982202","state":["exists","up"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3608171775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3608171775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3608171775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3608171775}]},"public_addr":"172.21.15.38:6813/3608171775","cluster_addr":"172.21.15.38:6814/3608171775","heartbeat_back_addr":"172.21.15.38:6816/3608171775","heartbeat_front_addr":"172.21.15.38:6815/3608171775","state":["exists","up"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2580659844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2580659844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":2580659844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2580659844}]},"public_addr":"172.21.15.118:6804/2580659844","cluster_addr":"172.21.15.118:6805/2580659844","heartbeat_back_addr":"172.21.15.118:6807/2580659844","heartbeat_front_addr":"172.21.15.118:6806/2580659844","state":["exists","up"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2352891695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":2352891695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":2352891695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2352891695}]},"public_addr":"172.21.15.154:6812/2352891695","cluster_addr":"172.21.15.154:6813/2352891695","heartbeat_back_addr":"172.21.15.154:6815/2352891695","heartbeat_front_addr":"172.21.15.154:6814/2352891695","state":["exists","up"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2304836889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":2304836889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":2304836889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2304836889}]},"public_addr":"172.21.15.204:6800/2304836889","cluster_addr":"172.21.15.204:6801/2304836889","heartbeat_back_addr":"172.21.15.204:6803/2304836889","heartbeat_front_addr":"172.21.15.204:6802/2304836889","state":["exists","up"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":1688419254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1688419254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1688419254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":1688419254}]},"public_addr":"172.21.15.38:6809/1688419254","cluster_addr":"172.21.15.38:6810/1688419254","heartbeat_back_addr":"172.21.15.38:6812/1688419254","heartbeat_front_addr":"172.21.15.38:6811/1688419254","state":["exists","up"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":3727425991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":3727425991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":3727425991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":3727425991}]},"public_addr":"172.21.15.118:6812/3727425991","cluster_addr":"172.21.15.118:6813/3727425991","heartbeat_back_addr":"172.21.15.118:6815/3727425991","heartbeat_front_addr":"172.21.15.118:6814/3727425991","state":["exists","up"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":538351706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":538351706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":538351706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":538351706}]},"public_addr":"172.21.15.154:6804/538351706","cluster_addr":"172.21.15.154:6805/538351706","heartbeat_back_addr":"172.21.15.154:6807/538351706","heartbeat_front_addr":"172.21.15.154:6806/538351706","state":["exists","up"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":501019151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":501019151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":501019151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":501019151}]},"public_addr":"172.21.15.204:6812/501019151","cluster_addr":"172.21.15.204:6813/501019151","heartbeat_back_addr":"172.21.15.204:6815/501019151","heartbeat_front_addr":"172.21.15.204:6814/501019151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.680732+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.915157+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.198442+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.248734+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.448334+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.873463+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.149077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.739636+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.850336+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.655711+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.092457+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.246352+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.027609+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.369804+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.979821+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.174959+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:32:09.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T11:32:09.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T11:32:09.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T11:32:09.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T11:32:09.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T11:32:09.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T11:32:09.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T11:32:09.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T11:32:09.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T11:32:09.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T11:32:09.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T11:32:09.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T11:32:09.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T11:32:09.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T11:32:09.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T11:32:09.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T11:32:10.082 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:10.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout:98784247812 2024-09-13T11:32:10.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:10.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:10.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:10.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:32:10.201 INFO:teuthology.orchestra.run.smithi038.stdout:90194313219 2024-09-13T11:32:10.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T11:32:10.220 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:10.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T11:32:10.222 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:10.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:32:10.240 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:10.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:32:10.241 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:10.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T11:32:10.242 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:10.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T11:32:10.254 INFO:teuthology.orchestra.run.smithi038.stdout:90194313219 2024-09-13T11:32:10.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:32:10.262 INFO:teuthology.orchestra.run.smithi038.stdout:98784247813 2024-09-13T11:32:10.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T11:32:10.274 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:10.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T11:32:10.282 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:10.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:32:10.287 INFO:teuthology.orchestra.run.smithi038.stdout:98784247812 2024-09-13T11:32:10.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T11:32:11.160 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:11.167 INFO:teuthology.orchestra.run.smithi038.stdout:94489280514 2024-09-13T11:32:11.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.6 2024-09-13T11:32:11.203 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.8 2024-09-13T11:32:11.239 INFO:teuthology.orchestra.run.smithi038.stdout:98784247810 2024-09-13T11:32:11.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247810 for osd.7 2024-09-13T11:32:11.295 INFO:teuthology.orchestra.run.smithi038.stdout:94489280514 2024-09-13T11:32:11.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.4 2024-09-13T11:32:11.387 INFO:teuthology.orchestra.run.smithi038.stdout:94489280514 2024-09-13T11:32:11.418 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:11.432 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.1 2024-09-13T11:32:11.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.12 2024-09-13T11:32:11.446 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.463 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:11.463 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:11.466 INFO:teuthology.orchestra.run.smithi038.stdout:98784247812 2024-09-13T11:32:11.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.13 2024-09-13T11:32:11.494 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2024-09-13T11:32:11.499 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.0 2024-09-13T11:32:11.500 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.500 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.501 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:11.505 INFO:teuthology.orchestra.run.smithi038.stdout:90194313219 2024-09-13T11:32:11.506 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:11.511 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:11.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.14 2024-09-13T11:32:11.516 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.518 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:11.520 INFO:teuthology.orchestra.run.smithi038.stdout:90194313219 2024-09-13T11:32:11.523 INFO:teuthology.orchestra.run.smithi038.stdout:98784247813 2024-09-13T11:32:11.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.11 2024-09-13T11:32:11.526 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.9 2024-09-13T11:32:11.529 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.10 2024-09-13T11:32:11.534 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.15 2024-09-13T11:32:11.542 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.2 2024-09-13T11:32:11.543 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:11.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.5 2024-09-13T11:32:11.544 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:12.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T11:32:12.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T11:32:12.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T11:32:12.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:32:12.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:32:12.602 INFO:teuthology.orchestra.run.smithi038.stdout:94489280516 2024-09-13T11:32:12.607 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:12.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.6 2024-09-13T11:32:12.621 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:12.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.8 2024-09-13T11:32:12.624 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:12.639 INFO:teuthology.orchestra.run.smithi038.stdout:98784247812 2024-09-13T11:32:12.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.7 2024-09-13T11:32:12.654 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:12.666 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:12.680 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.4 2024-09-13T11:32:12.680 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:12.805 INFO:teuthology.orchestra.run.smithi038.stdout:94489280515 2024-09-13T11:32:12.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.1 2024-09-13T11:32:12.818 DEBUG:teuthology.parallel:result is None 2024-09-13T11:32:12.819 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:32:12.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:32:13.118 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:13.119 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:32:13.131 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T11:32:12.928386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":455000,"kb_used_data":5976,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509494440,"statfs":{"total":1546188226560,"available":1545722306560,"internally_reserved":0,"allocated":6119424,"data_stored":3418329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25522,"internal_metadata":459512910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624990+0000","last_change":"2024-09-13T11:32:08.624990+0000","last_active":"2024-09-13T11:32:08.624990+0000","last_peered":"2024-09-13T11:32:08.624990+0000","last_clean":"2024-09-13T11:32:08.624990+0000","last_became_active":"2024-09-13T11:32:06.351947+0000","last_became_peered":"2024-09-13T11:32:06.351947+0000","last_unstale":"2024-09-13T11:32:08.624990+0000","last_undegraded":"2024-09-13T11:32:08.624990+0000","last_fullsized":"2024-09-13T11:32:08.624990+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:21.025838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,6],"acting":[15,13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624828+0000","last_change":"2024-09-13T11:32:08.625099+0000","last_active":"2024-09-13T11:32:08.624828+0000","last_peered":"2024-09-13T11:32:08.624828+0000","last_clean":"2024-09-13T11:32:08.624828+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:32:08.624828+0000","last_undegraded":"2024-09-13T11:32:08.624828+0000","last_fullsized":"2024-09-13T11:32:08.624828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:31.093378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.579764+0000","last_change":"2024-09-13T11:32:08.579949+0000","last_active":"2024-09-13T11:32:08.579764+0000","last_peered":"2024-09-13T11:32:08.579764+0000","last_clean":"2024-09-13T11:32:08.579764+0000","last_became_active":"2024-09-13T11:32:06.731632+0000","last_became_peered":"2024-09-13T11:32:06.731632+0000","last_unstale":"2024-09-13T11:32:08.579764+0000","last_undegraded":"2024-09-13T11:32:08.579764+0000","last_fullsized":"2024-09-13T11:32:08.579764+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:12.761704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624646+0000","last_change":"2024-09-13T11:32:08.624950+0000","last_active":"2024-09-13T11:32:08.624646+0000","last_peered":"2024-09-13T11:32:08.624646+0000","last_clean":"2024-09-13T11:32:08.624646+0000","last_became_active":"2024-09-13T11:32:06.755851+0000","last_became_peered":"2024-09-13T11:32:06.755851+0000","last_unstale":"2024-09-13T11:32:08.624646+0000","last_undegraded":"2024-09-13T11:32:08.624646+0000","last_fullsized":"2024-09-13T11:32:08.624646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.022985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.380746+0000","last_change":"2024-09-13T11:32:08.380746+0000","last_active":"2024-09-13T11:32:08.380746+0000","last_peered":"2024-09-13T11:32:08.380746+0000","last_clean":"2024-09-13T11:32:08.380746+0000","last_became_active":"2024-09-13T11:32:06.754189+0000","last_became_peered":"2024-09-13T11:32:06.754189+0000","last_unstale":"2024-09-13T11:32:08.380746+0000","last_undegraded":"2024-09-13T11:32:08.380746+0000","last_fullsized":"2024-09-13T11:32:08.380746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:30.208840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624640+0000","last_change":"2024-09-13T11:32:08.624640+0000","last_active":"2024-09-13T11:32:08.624640+0000","last_peered":"2024-09-13T11:32:08.624640+0000","last_clean":"2024-09-13T11:32:08.624640+0000","last_became_active":"2024-09-13T11:32:06.351932+0000","last_became_peered":"2024-09-13T11:32:06.351932+0000","last_unstale":"2024-09-13T11:32:08.624640+0000","last_undegraded":"2024-09-13T11:32:08.624640+0000","last_fullsized":"2024-09-13T11:32:08.624640+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:08.555102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.377104+0000","last_change":"2024-09-13T11:32:08.377316+0000","last_active":"2024-09-13T11:32:08.377104+0000","last_peered":"2024-09-13T11:32:08.377104+0000","last_clean":"2024-09-13T11:32:08.377104+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:32:08.377104+0000","last_undegraded":"2024-09-13T11:32:08.377104+0000","last_fullsized":"2024-09-13T11:32:08.377104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:26.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.381943+0000","last_change":"2024-09-13T11:32:08.382182+0000","last_active":"2024-09-13T11:32:08.381943+0000","last_peered":"2024-09-13T11:32:08.381943+0000","last_clean":"2024-09-13T11:32:08.381943+0000","last_became_active":"2024-09-13T11:32:06.347877+0000","last_became_peered":"2024-09-13T11:32:06.347877+0000","last_unstale":"2024-09-13T11:32:08.381943+0000","last_undegraded":"2024-09-13T11:32:08.381943+0000","last_fullsized":"2024-09-13T11:32:08.381943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:10.391363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045723999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":156,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624103+0000","last_change":"2024-09-13T11:32:04.646107+0000","last_active":"2024-09-13T11:32:08.624103+0000","last_peered":"2024-09-13T11:32:08.624103+0000","last_clean":"2024-09-13T11:32:08.624103+0000","last_became_active":"2024-09-13T11:32:04.643836+0000","last_became_peered":"2024-09-13T11:32:04.643836+0000","last_unstale":"2024-09-13T11:32:08.624103+0000","last_undegraded":"2024-09-13T11:32:08.624103+0000","last_fullsized":"2024-09-13T11:32:08.624103+0000","mapping_epoch":23,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:32:03.312547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:03.530977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":868352,"data_stored":693335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,7,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:32:13.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:32:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:13.427 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:32:13.439 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-13T11:32:12.928386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":455000,"kb_used_data":5976,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509494440,"statfs":{"total":1546188226560,"available":1545722306560,"internally_reserved":0,"allocated":6119424,"data_stored":3418329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25522,"internal_metadata":459512910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.567689"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624990+0000","last_change":"2024-09-13T11:32:08.624990+0000","last_active":"2024-09-13T11:32:08.624990+0000","last_peered":"2024-09-13T11:32:08.624990+0000","last_clean":"2024-09-13T11:32:08.624990+0000","last_became_active":"2024-09-13T11:32:06.351947+0000","last_became_peered":"2024-09-13T11:32:06.351947+0000","last_unstale":"2024-09-13T11:32:08.624990+0000","last_undegraded":"2024-09-13T11:32:08.624990+0000","last_fullsized":"2024-09-13T11:32:08.624990+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:21.025838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,6],"acting":[15,13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624828+0000","last_change":"2024-09-13T11:32:08.625099+0000","last_active":"2024-09-13T11:32:08.624828+0000","last_peered":"2024-09-13T11:32:08.624828+0000","last_clean":"2024-09-13T11:32:08.624828+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:32:08.624828+0000","last_undegraded":"2024-09-13T11:32:08.624828+0000","last_fullsized":"2024-09-13T11:32:08.624828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:31.093378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.579764+0000","last_change":"2024-09-13T11:32:08.579949+0000","last_active":"2024-09-13T11:32:08.579764+0000","last_peered":"2024-09-13T11:32:08.579764+0000","last_clean":"2024-09-13T11:32:08.579764+0000","last_became_active":"2024-09-13T11:32:06.731632+0000","last_became_peered":"2024-09-13T11:32:06.731632+0000","last_unstale":"2024-09-13T11:32:08.579764+0000","last_undegraded":"2024-09-13T11:32:08.579764+0000","last_fullsized":"2024-09-13T11:32:08.579764+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:12.761704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624646+0000","last_change":"2024-09-13T11:32:08.624950+0000","last_active":"2024-09-13T11:32:08.624646+0000","last_peered":"2024-09-13T11:32:08.624646+0000","last_clean":"2024-09-13T11:32:08.624646+0000","last_became_active":"2024-09-13T11:32:06.755851+0000","last_became_peered":"2024-09-13T11:32:06.755851+0000","last_unstale":"2024-09-13T11:32:08.624646+0000","last_undegraded":"2024-09-13T11:32:08.624646+0000","last_fullsized":"2024-09-13T11:32:08.624646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.022985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.380746+0000","last_change":"2024-09-13T11:32:08.380746+0000","last_active":"2024-09-13T11:32:08.380746+0000","last_peered":"2024-09-13T11:32:08.380746+0000","last_clean":"2024-09-13T11:32:08.380746+0000","last_became_active":"2024-09-13T11:32:06.754189+0000","last_became_peered":"2024-09-13T11:32:06.754189+0000","last_unstale":"2024-09-13T11:32:08.380746+0000","last_undegraded":"2024-09-13T11:32:08.380746+0000","last_fullsized":"2024-09-13T11:32:08.380746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:30.208840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624640+0000","last_change":"2024-09-13T11:32:08.624640+0000","last_active":"2024-09-13T11:32:08.624640+0000","last_peered":"2024-09-13T11:32:08.624640+0000","last_clean":"2024-09-13T11:32:08.624640+0000","last_became_active":"2024-09-13T11:32:06.351932+0000","last_became_peered":"2024-09-13T11:32:06.351932+0000","last_unstale":"2024-09-13T11:32:08.624640+0000","last_undegraded":"2024-09-13T11:32:08.624640+0000","last_fullsized":"2024-09-13T11:32:08.624640+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:08.555102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.377104+0000","last_change":"2024-09-13T11:32:08.377316+0000","last_active":"2024-09-13T11:32:08.377104+0000","last_peered":"2024-09-13T11:32:08.377104+0000","last_clean":"2024-09-13T11:32:08.377104+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:32:08.377104+0000","last_undegraded":"2024-09-13T11:32:08.377104+0000","last_fullsized":"2024-09-13T11:32:08.377104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:26.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.381943+0000","last_change":"2024-09-13T11:32:08.382182+0000","last_active":"2024-09-13T11:32:08.381943+0000","last_peered":"2024-09-13T11:32:08.381943+0000","last_clean":"2024-09-13T11:32:08.381943+0000","last_became_active":"2024-09-13T11:32:06.347877+0000","last_became_peered":"2024-09-13T11:32:06.347877+0000","last_unstale":"2024-09-13T11:32:08.381943+0000","last_undegraded":"2024-09-13T11:32:08.381943+0000","last_fullsized":"2024-09-13T11:32:08.381943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:10.391363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045723999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":156,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624103+0000","last_change":"2024-09-13T11:32:04.646107+0000","last_active":"2024-09-13T11:32:08.624103+0000","last_peered":"2024-09-13T11:32:08.624103+0000","last_clean":"2024-09-13T11:32:08.624103+0000","last_became_active":"2024-09-13T11:32:04.643836+0000","last_became_peered":"2024-09-13T11:32:04.643836+0000","last_unstale":"2024-09-13T11:32:08.624103+0000","last_undegraded":"2024-09-13T11:32:08.624103+0000","last_fullsized":"2024-09-13T11:32:08.624103+0000","mapping_epoch":23,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:32:03.312547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:03.530977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":868352,"data_stored":693335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,7,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:32:13.441 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:32:13.441 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T11:32:13.441 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T11:32:13.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T11:32:13.775 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:13.775 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T11:32:13.788 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T11:32:13.789 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T11:32:13.799 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T11:32:13.800 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:32:13.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "active": true, 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_duration": "0:00:00.000303", 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_started": "Fri Sep 13 11:31:54 2024", 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "upmap", 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "no_optimization_needed": false, 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "optimize_result": "No pools available", 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "plans": [] 2024-09-13T11:32:14.098 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:14.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-09-13T11:32:15.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-09-13T11:32:15.606 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-13T11:32:15.616 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 2, 'chance_pgpnum_fix': 1, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-13T11:32:15.616 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore odsync write': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 0.5}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:32:15.616 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 2, 'chance_pgpnum_fix': 1, 'min_in': 4, 'thrash_hosts': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-13T11:32:15.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-13T11:32:15.938 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-13T11:32:15.938 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:32:15.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:32:16.235 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:32:16.248 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T11:32:15.259146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":455000,"kb_used_data":5976,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509494440,"statfs":{"total":1546188226560,"available":1545722306560,"internally_reserved":0,"allocated":6119424,"data_stored":3418329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25522,"internal_metadata":459512910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.898449"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624990+0000","last_change":"2024-09-13T11:32:08.624990+0000","last_active":"2024-09-13T11:32:08.624990+0000","last_peered":"2024-09-13T11:32:08.624990+0000","last_clean":"2024-09-13T11:32:08.624990+0000","last_became_active":"2024-09-13T11:32:06.351947+0000","last_became_peered":"2024-09-13T11:32:06.351947+0000","last_unstale":"2024-09-13T11:32:08.624990+0000","last_undegraded":"2024-09-13T11:32:08.624990+0000","last_fullsized":"2024-09-13T11:32:08.624990+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:21.025838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,6],"acting":[15,13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624828+0000","last_change":"2024-09-13T11:32:08.625099+0000","last_active":"2024-09-13T11:32:08.624828+0000","last_peered":"2024-09-13T11:32:08.624828+0000","last_clean":"2024-09-13T11:32:08.624828+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:32:08.624828+0000","last_undegraded":"2024-09-13T11:32:08.624828+0000","last_fullsized":"2024-09-13T11:32:08.624828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:31.093378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.579764+0000","last_change":"2024-09-13T11:32:08.579949+0000","last_active":"2024-09-13T11:32:08.579764+0000","last_peered":"2024-09-13T11:32:08.579764+0000","last_clean":"2024-09-13T11:32:08.579764+0000","last_became_active":"2024-09-13T11:32:06.731632+0000","last_became_peered":"2024-09-13T11:32:06.731632+0000","last_unstale":"2024-09-13T11:32:08.579764+0000","last_undegraded":"2024-09-13T11:32:08.579764+0000","last_fullsized":"2024-09-13T11:32:08.579764+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:12.761704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624646+0000","last_change":"2024-09-13T11:32:08.624950+0000","last_active":"2024-09-13T11:32:08.624646+0000","last_peered":"2024-09-13T11:32:08.624646+0000","last_clean":"2024-09-13T11:32:08.624646+0000","last_became_active":"2024-09-13T11:32:06.755851+0000","last_became_peered":"2024-09-13T11:32:06.755851+0000","last_unstale":"2024-09-13T11:32:08.624646+0000","last_undegraded":"2024-09-13T11:32:08.624646+0000","last_fullsized":"2024-09-13T11:32:08.624646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.022985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.380746+0000","last_change":"2024-09-13T11:32:08.380746+0000","last_active":"2024-09-13T11:32:08.380746+0000","last_peered":"2024-09-13T11:32:08.380746+0000","last_clean":"2024-09-13T11:32:08.380746+0000","last_became_active":"2024-09-13T11:32:06.754189+0000","last_became_peered":"2024-09-13T11:32:06.754189+0000","last_unstale":"2024-09-13T11:32:08.380746+0000","last_undegraded":"2024-09-13T11:32:08.380746+0000","last_fullsized":"2024-09-13T11:32:08.380746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:30.208840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624640+0000","last_change":"2024-09-13T11:32:08.624640+0000","last_active":"2024-09-13T11:32:08.624640+0000","last_peered":"2024-09-13T11:32:08.624640+0000","last_clean":"2024-09-13T11:32:08.624640+0000","last_became_active":"2024-09-13T11:32:06.351932+0000","last_became_peered":"2024-09-13T11:32:06.351932+0000","last_unstale":"2024-09-13T11:32:08.624640+0000","last_undegraded":"2024-09-13T11:32:08.624640+0000","last_fullsized":"2024-09-13T11:32:08.624640+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:08.555102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.377104+0000","last_change":"2024-09-13T11:32:08.377316+0000","last_active":"2024-09-13T11:32:08.377104+0000","last_peered":"2024-09-13T11:32:08.377104+0000","last_clean":"2024-09-13T11:32:08.377104+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:32:08.377104+0000","last_undegraded":"2024-09-13T11:32:08.377104+0000","last_fullsized":"2024-09-13T11:32:08.377104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:26.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.381943+0000","last_change":"2024-09-13T11:32:08.382182+0000","last_active":"2024-09-13T11:32:08.381943+0000","last_peered":"2024-09-13T11:32:08.381943+0000","last_clean":"2024-09-13T11:32:08.381943+0000","last_became_active":"2024-09-13T11:32:06.347877+0000","last_became_peered":"2024-09-13T11:32:06.347877+0000","last_unstale":"2024-09-13T11:32:08.381943+0000","last_undegraded":"2024-09-13T11:32:08.381943+0000","last_fullsized":"2024-09-13T11:32:08.381943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:10.391363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045723999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":156,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624103+0000","last_change":"2024-09-13T11:32:04.646107+0000","last_active":"2024-09-13T11:32:08.624103+0000","last_peered":"2024-09-13T11:32:08.624103+0000","last_clean":"2024-09-13T11:32:08.624103+0000","last_became_active":"2024-09-13T11:32:04.643836+0000","last_became_peered":"2024-09-13T11:32:04.643836+0000","last_unstale":"2024-09-13T11:32:08.624103+0000","last_undegraded":"2024-09-13T11:32:08.624103+0000","last_fullsized":"2024-09-13T11:32:08.624103+0000","mapping_epoch":23,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:32:03.312547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:03.530977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247813,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":868352,"data_stored":693335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,7,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:32:16.249 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:32:16.547 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:16.547 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:32:16.559 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T11:32:15.259146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":455000,"kb_used_data":5976,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509494440,"statfs":{"total":1546188226560,"available":1545722306560,"internally_reserved":0,"allocated":6119424,"data_stored":3418329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25522,"internal_metadata":459512910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.898449"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624990+0000","last_change":"2024-09-13T11:32:08.624990+0000","last_active":"2024-09-13T11:32:08.624990+0000","last_peered":"2024-09-13T11:32:08.624990+0000","last_clean":"2024-09-13T11:32:08.624990+0000","last_became_active":"2024-09-13T11:32:06.351947+0000","last_became_peered":"2024-09-13T11:32:06.351947+0000","last_unstale":"2024-09-13T11:32:08.624990+0000","last_undegraded":"2024-09-13T11:32:08.624990+0000","last_fullsized":"2024-09-13T11:32:08.624990+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:21.025838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,6],"acting":[15,13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624828+0000","last_change":"2024-09-13T11:32:08.625099+0000","last_active":"2024-09-13T11:32:08.624828+0000","last_peered":"2024-09-13T11:32:08.624828+0000","last_clean":"2024-09-13T11:32:08.624828+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:32:08.624828+0000","last_undegraded":"2024-09-13T11:32:08.624828+0000","last_fullsized":"2024-09-13T11:32:08.624828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:31.093378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.579764+0000","last_change":"2024-09-13T11:32:08.579949+0000","last_active":"2024-09-13T11:32:08.579764+0000","last_peered":"2024-09-13T11:32:08.579764+0000","last_clean":"2024-09-13T11:32:08.579764+0000","last_became_active":"2024-09-13T11:32:06.731632+0000","last_became_peered":"2024-09-13T11:32:06.731632+0000","last_unstale":"2024-09-13T11:32:08.579764+0000","last_undegraded":"2024-09-13T11:32:08.579764+0000","last_fullsized":"2024-09-13T11:32:08.579764+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:12.761704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624646+0000","last_change":"2024-09-13T11:32:08.624950+0000","last_active":"2024-09-13T11:32:08.624646+0000","last_peered":"2024-09-13T11:32:08.624646+0000","last_clean":"2024-09-13T11:32:08.624646+0000","last_became_active":"2024-09-13T11:32:06.755851+0000","last_became_peered":"2024-09-13T11:32:06.755851+0000","last_unstale":"2024-09-13T11:32:08.624646+0000","last_undegraded":"2024-09-13T11:32:08.624646+0000","last_fullsized":"2024-09-13T11:32:08.624646+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.022985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.380746+0000","last_change":"2024-09-13T11:32:08.380746+0000","last_active":"2024-09-13T11:32:08.380746+0000","last_peered":"2024-09-13T11:32:08.380746+0000","last_clean":"2024-09-13T11:32:08.380746+0000","last_became_active":"2024-09-13T11:32:06.754189+0000","last_became_peered":"2024-09-13T11:32:06.754189+0000","last_unstale":"2024-09-13T11:32:08.380746+0000","last_undegraded":"2024-09-13T11:32:08.380746+0000","last_fullsized":"2024-09-13T11:32:08.380746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:30.208840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624640+0000","last_change":"2024-09-13T11:32:08.624640+0000","last_active":"2024-09-13T11:32:08.624640+0000","last_peered":"2024-09-13T11:32:08.624640+0000","last_clean":"2024-09-13T11:32:08.624640+0000","last_became_active":"2024-09-13T11:32:06.351932+0000","last_became_peered":"2024-09-13T11:32:06.351932+0000","last_unstale":"2024-09-13T11:32:08.624640+0000","last_undegraded":"2024-09-13T11:32:08.624640+0000","last_fullsized":"2024-09-13T11:32:08.624640+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:08.555102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.377104+0000","last_change":"2024-09-13T11:32:08.377316+0000","last_active":"2024-09-13T11:32:08.377104+0000","last_peered":"2024-09-13T11:32:08.377104+0000","last_clean":"2024-09-13T11:32:08.377104+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:32:08.377104+0000","last_undegraded":"2024-09-13T11:32:08.377104+0000","last_fullsized":"2024-09-13T11:32:08.377104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:26.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.381943+0000","last_change":"2024-09-13T11:32:08.382182+0000","last_active":"2024-09-13T11:32:08.381943+0000","last_peered":"2024-09-13T11:32:08.381943+0000","last_clean":"2024-09-13T11:32:08.381943+0000","last_became_active":"2024-09-13T11:32:06.347877+0000","last_became_peered":"2024-09-13T11:32:06.347877+0000","last_unstale":"2024-09-13T11:32:08.381943+0000","last_undegraded":"2024-09-13T11:32:08.381943+0000","last_fullsized":"2024-09-13T11:32:08.381943+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:10.391363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045723999999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"25'112","reported_seq":156,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-13T11:32:08.624103+0000","last_change":"2024-09-13T11:32:04.646107+0000","last_active":"2024-09-13T11:32:08.624103+0000","last_peered":"2024-09-13T11:32:08.624103+0000","last_clean":"2024-09-13T11:32:08.624103+0000","last_became_active":"2024-09-13T11:32:04.643836+0000","last_became_peered":"2024-09-13T11:32:04.643836+0000","last_unstale":"2024-09-13T11:32:08.624103+0000","last_undegraded":"2024-09-13T11:32:08.624103+0000","last_fullsized":"2024-09-13T11:32:08.624103+0000","mapping_epoch":23,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:32:03.312547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:03.530977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247813,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":864256,"data_stored":693316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":868352,"data_stored":693335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,7,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":102948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,9,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:32:16.560 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:32:16.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-13T11:32:16.859 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 29 2024-09-13T11:32:16.859 INFO:teuthology.orchestra.run.smithi038.stdout:fsid b332d022-c5e6-4682-9d3d-2b6994cd0f20 2024-09-13T11:32:16.859 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-09-13T11:31:50.491934+0000 2024-09-13T11:32:16.859 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-09-13T11:32:15.255822+0000 2024-09-13T11:32:16.859 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 7 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client luminous 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client jewel 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release squid 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.79 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.00 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 16 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6805/2677506544 v2:172.21.15.38:6806/2677506544 exists,up 5a4d0d43-6562-441e-925e-4c992ee7b335 2024-09-13T11:32:16.860 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6808/3561806747 v2:172.21.15.118:6809/3561806747 exists,up 80665caa-6933-4f89-b454-128057e9ad92 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6800/4217282489 v2:172.21.15.154:6801/4217282489 exists,up eb99fd51-a064-4311-932e-8b6d555409af 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6804/2326505959 v2:172.21.15.204:6805/2326505959 exists,up 3d840ed9-a410-4bcb-b0f0-70fafdf69842 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6801/3107658120 v2:172.21.15.38:6802/3107658120 exists,up 0c01b7d7-1309-40fa-913a-97d2a8499293 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6800/3800782738 v2:172.21.15.118:6801/3800782738 exists,up b4cec928-1764-4fe0-8c5d-870d8a0c15c7 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6808/3986362716 v2:172.21.15.154:6809/3986362716 exists,up 89f6d810-2afc-42b0-b989-052218b6887c 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6808/1978982202 v2:172.21.15.204:6809/1978982202 exists,up ccfee49f-f394-4de0-8e30-853f38e31232 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6813/3608171775 v2:172.21.15.38:6814/3608171775 exists,up 03b1c505-1b92-4daa-9da4-c254be2ba792 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6804/2580659844 v2:172.21.15.118:6805/2580659844 exists,up 8fbee7a0-2ee6-4551-bf06-4e013e413fb4 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6812/2352891695 v2:172.21.15.154:6813/2352891695 exists,up 0808cb41-f62e-49b0-935d-616bea1a4db5 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6800/2304836889 v2:172.21.15.204:6801/2304836889 exists,up 427dd574-f2a8-49e5-a32a-cfda023a514a 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6809/1688419254 v2:172.21.15.38:6810/1688419254 exists,up 17942689-1130-4b6b-9f98-1605a03731b1 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6812/3727425991 v2:172.21.15.118:6813/3727425991 exists,up 62161cab-6d14-4571-971d-217601e38fbc 2024-09-13T11:32:16.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6804/538351706 v2:172.21.15.154:6805/538351706 exists,up 559ef141-27db-452d-9577-a6135421f64f 2024-09-13T11:32:16.862 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6812/501019151 v2:172.21.15.204:6813/501019151 exists,up 98f9ceba-79df-410a-9788-393057e79c30 2024-09-13T11:32:16.873 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6805/2677506544 v2:172.21.15.38:6806/2677506544 exists,up 5a4d0d43-6562-441e-925e-4c992ee7b335', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6808/3561806747 v2:172.21.15.118:6809/3561806747 exists,up 80665caa-6933-4f89-b454-128057e9ad92', 'osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6800/4217282489 v2:172.21.15.154:6801/4217282489 exists,up eb99fd51-a064-4311-932e-8b6d555409af', 'osd.3 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6804/2326505959 v2:172.21.15.204:6805/2326505959 exists,up 3d840ed9-a410-4bcb-b0f0-70fafdf69842', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6801/3107658120 v2:172.21.15.38:6802/3107658120 exists,up 0c01b7d7-1309-40fa-913a-97d2a8499293', 'osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6800/3800782738 v2:172.21.15.118:6801/3800782738 exists,up b4cec928-1764-4fe0-8c5d-870d8a0c15c7', 'osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6808/3986362716 v2:172.21.15.154:6809/3986362716 exists,up 89f6d810-2afc-42b0-b989-052218b6887c', 'osd.7 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6808/1978982202 v2:172.21.15.204:6809/1978982202 exists,up ccfee49f-f394-4de0-8e30-853f38e31232', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6813/3608171775 v2:172.21.15.38:6814/3608171775 exists,up 03b1c505-1b92-4daa-9da4-c254be2ba792', 'osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6804/2580659844 v2:172.21.15.118:6805/2580659844 exists,up 8fbee7a0-2ee6-4551-bf06-4e013e413fb4', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6812/2352891695 v2:172.21.15.154:6813/2352891695 exists,up 0808cb41-f62e-49b0-935d-616bea1a4db5', 'osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6800/2304836889 v2:172.21.15.204:6801/2304836889 exists,up 427dd574-f2a8-49e5-a32a-cfda023a514a', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v2:172.21.15.38:6809/1688419254 v2:172.21.15.38:6810/1688419254 exists,up 17942689-1130-4b6b-9f98-1605a03731b1', 'osd.13 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6812/3727425991 v2:172.21.15.118:6813/3727425991 exists,up 62161cab-6d14-4571-971d-217601e38fbc', 'osd.14 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6804/538351706 v2:172.21.15.154:6805/538351706 exists,up 559ef141-27db-452d-9577-a6135421f64f', 'osd.15 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v2:172.21.15.204:6812/501019151 v2:172.21.15.204:6813/501019151 exists,up 98f9ceba-79df-410a-9788-393057e79c30'] 2024-09-13T11:32:16.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-13T11:32:16.976 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mon.a", 2024-09-13T11:32:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster": "ceph", 2024-09-13T11:32:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-13T11:32:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket_mode": "", 2024-09-13T11:32:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_client_required": "cephx, none", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_cluster_required": "cephx", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_debug": "true", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_required": "cephx", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_supported": "cephx", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio": "true", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_poll_ms": "250", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_reap_max": "16", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_async_discard_threads": "0", 2024-09-13T11:32:16.977 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_block_size": "4096", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio": "false", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_inflight_ios": "false", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_enable_discard": "false", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry": "3", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash": "0", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring": "false", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_hipri": "false", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-13T11:32:16.978 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_type": "", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_alloc_size": "1048576", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_allocator": "hybrid", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_buffered_io": "true", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_for_zeros": "false", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_compact_log_sync": "false", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-13T11:32:16.979 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_log_runway": "4194304", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_prefetch": "1048576", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_flush_size": "524288", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_log_runway": "1048576", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery": "false", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_sync_write": "false", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-13T11:32:16.980 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocation_from_file": "true", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocator": "hybrid", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_async_db_compaction": "true", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bdev_label_multi": "true", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-13T11:32:16.981 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_create": "true", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_create": "false", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_path": "", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_size": "0", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_path": "", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_preallocate_file": "false", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_size": "107374182400", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_create": "false", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_path": "", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_size": "100663296", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs": "true", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-13T11:32:16.982 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune": "true", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size": "0", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-13T11:32:16.983 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_type": "2q", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_clone_cow": "true", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_mode": "none", 2024-09-13T11:32:16.984 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_csum_type": "crc32c", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_freelist": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-13T11:32:16.985 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_small_allocations": "0", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_read": "true", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_write": "false", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-13T11:32:16.986 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fail_eio": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-13T11:32:16.987 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_ignore_data_csum": "false", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_op_age": "5.000000", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_alloc_size": "0", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size": "0", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-13T11:32:16.988 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_deferred_txc": "32", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size": "0", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_nid_prealloc": "1024", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_retry_disk_reads": "3", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cf": "true", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-13T11:32:16.989 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_mem": "512", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-13T11:32:16.990 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_tracing": "false", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_write_v2": "false", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_write_v2_random": "false", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_zero_block_detection": "false", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-13T11:32:16.991 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-13T11:32:16.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_signatures": "false", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_version": "2", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_signatures": "false", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_version": "2", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_signatures": "false", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_version": "2", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_sign_messages": "true", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "chdir": "", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_acl_type": "", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_asio_thread_count": "2", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_mid": "0.750000", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_size": "16384", 2024-09-13T11:32:16.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_caps_release_delay": "5", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_check_pool_perm": "true", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_force_sync_read": "false", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_getattr_caps": "false", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_inject_features": "", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_inject_tick_delay": "0", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_remount": "false", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_dirsize_rbytes": "true", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_force_lazyio": "false", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_fs": "", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_release_failure": "false", 2024-09-13T11:32:16.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_inline_size": "4096", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mds_namespace": "", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_metadata": "", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_gid": "-1", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_timeout": "300", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_uid": "-1", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mountpoint": "/", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_notify_timeout": "10", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc": "true", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty": "104857600", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_objects": "1000", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_size": "209715200", 2024-09-13T11:32:16.995 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_target_dirty": "8388608", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_permissions": "true", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota": "true", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota_df": "true", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_bytes": "0", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_periods": "4", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_min": "131072", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_reconnect_stale": "false", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_shutdown_timeout": "30", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_snapdir": ".snap", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_tick_interval": "1", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_trace": "", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_try_dentry_invalidate": "false", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_faked_inos": "false", 2024-09-13T11:32:16.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_random_mds": "false", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog": "false", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_port": "12201", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_monitors": "default=true", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog": "false", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_level": "info", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "-", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network": "", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network_interface": "", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_isal": "false", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_level": "5", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_winsize": "-15", 2024-09-13T11:32:16.997 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zstd_level": "1", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_op_num_threads": "6", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_stat_interval": "0", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_seastar_cpu_cores": "", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_seastar_num_threads": "0", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook": "", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook_timeout": "10", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "daemonize": "false", 2024-09-13T11:32:16.998 INFO:teuthology.orchestra.run.smithi038.stdout: "daos_pool": "tank", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_alienstore": "0/5", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_allow_any_pool_priority": "false", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok": "1/5", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok_assert_abort": "false", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asserts_on_shutdown": "true", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_auth": "20/20", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bdev": "1/3", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluefs": "1/5", 2024-09-13T11:32:16.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluestore": "1/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_buffer": "0/1", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ceph_dedup": "0/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ceph_exporter": "1/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephfs_mirror": "0/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephsqlite": "0/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_client": "0/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_compressor": "1/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_context": "0/1", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crimson_interrupt": "0/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crush": "1/1", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crypto": "1/5", 2024-09-13T11:32:17.000 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cyanstore": "0/5", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_deliberately_leak_memory": "false", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_disable_randomized_ping": "false", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_dpdk": "1/5", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_eventtrace": "1/5", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filer": "0/1", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filestore": "1/3", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_finisher": "1/1", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_fuse": "1/5", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeat_testing_span": "0", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeatmap": "1/5", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_javaclient": "1/5", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journal": "1/3", 2024-09-13T11:32:17.001 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journaler": "0/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_kstore": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_lockdep": "0/1", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mclock": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_balancer": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_locker": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log_expire": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_migrator": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_quiesce": "3/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_memstore": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgr": "2/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgrc": "1/5", 2024-09-13T11:32:17.002 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mon": "20/20", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_monc": "0/10", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ms": "1/1", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_none": "0/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objclass": "0/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objectcacher": "0/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objecter": "0/1", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_optracker": "0/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_osd": "1/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_paxos": "20/20", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_perfcounter": "1/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_prioritycache": "1/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rados": "0/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd": "0/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_mirror": "0/5", 2024-09-13T11:32:17.003 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_pwl": "0/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_replay": "0/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_refs": "0/0", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_reserver": "1/1", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_access": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_datacache": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_dbstore": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_flight": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_notification": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_sync": "1/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rocksdb": "4/5", 2024-09-13T11:32:17.004 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_backref": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cache": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cleaner": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_device": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_epm": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_journal": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_lba": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_odata": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_omap": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_onode": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_t": "0/5", 2024-09-13T11:32:17.005 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_tm": "0/5", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_striper": "0/1", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_test": "0/5", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_throttle": "1/1", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_timer": "0/1", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_tp": "0/5", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_trace": "1/5", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "device_failure_prediction_mode": "none", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-13T11:32:17.006 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_graylog": "false", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_journald": "false", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_stderr": "true", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_syslog": "false", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event_tracing": "false", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_addr": "0.0.0.0", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_cert_file": "", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_http_port": "9926", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_key_file": "", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_prio_limit": "5", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sort_metrics": "true", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_stats_period": "5", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "fake_statfs_for_testing": "0", 2024-09-13T11:32:17.007 INFO:teuthology.orchestra.run.smithi038.stdout: "fatal_signal_handlers": "true", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_purge_ops": "10", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_truncate_ops": "128", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_apply_finisher_threads": "1", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_blackhole": "false", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_clone_range": "true", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_snap": "true", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_caller_concurrency": "10", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_collect_device_partition_information": "true", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_commit_timeout": "600.000000", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_inject_read_err": "false", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_omap_check": "false", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_verify_split": "false", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_dump_file": "", 2024-09-13T11:32:17.008 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fadvise": "true", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fail_eio": "true", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_shards": "16", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_size": "128", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap": "false", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap_threshold": "4096", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_inject_stall": "0", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_parallel": "false", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_trailing": "false", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_writeahead": "false", 2024-09-13T11:32:17.009 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_kill_at": "0", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs": "0", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size": "0", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-13T11:32:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_merge_threshold": "-10", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_odsync_write": "false", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend": "rocksdb", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend_path": "", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_timeout": "60", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_threads": "2", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_punch_hole": "false", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-13T11:32:17.011 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_ops": "50", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_seek_data_hole": "false", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc": "false", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_splice": "false", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_multiple": "2", 2024-09-13T11:32:17.012 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_rand_factor": "20", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_update_to": "1000", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_enable": "true", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-13T11:32:17.013 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-13T11:32:17.014 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-13T11:32:17.016 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_xfs_extsize": "false", 2024-09-13T11:32:17.016 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_zfs_snap": "false", 2024-09-13T11:32:17.016 INFO:teuthology.orchestra.run.smithi038.stdout: "fio_dir": "/tmp/fio", 2024-09-13T11:32:17.016 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "b332d022-c5e6-4682-9d3d-2b6994cd0f20", 2024-09-13T11:32:17.016 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_allow_other": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_atomic_o_trunc": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_big_writes": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_debug": "false", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_default_permissions": "false", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_disable_pagecache": "false", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_max_write": "0", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_multithreaded": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_require_active_mds": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_set_user_groups": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_move": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_read": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_write": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-13T11:32:17.017 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_use_invalidate_cb": "true", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_target_name": "ceph", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_file": "", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_inject_failure": "0", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_interval": "5", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "host": "smithi038", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-13T11:32:17.018 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "inject_early_sigterm": "false", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_agent_port": "6799", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_tracing_enable": "false", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_aio": "true", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_align_min_size": "65536", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_align": "true", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_size": "4096", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_dio": "true", 2024-09-13T11:32:17.019 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_discard": "false", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_force_aio": "false", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_ignore_corruption": "false", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_bytes": "10485760", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_entries": "100", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_replay_from": "0", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_write_header_frequency": "0", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_zero_on_create": "false", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prefetch_periods": "10", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prezero_periods": "5", 2024-09-13T11:32:17.020 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_write_head_interval": "15", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "keyfile": "", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_backend": "rocksdb", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_default_stripe_size": "65536", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount": "false", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_bytes": "67108864", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_ops": "512", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_nid_prealloc": "1024", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_onode_map_size": "1024", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_submit_transaction": "false", 2024-09-13T11:32:17.021 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_transaction": "false", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "librados_thread_count": "2", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep": "false", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep_force_backtrace": "false", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_coarse_timestamps": "true", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_flush_on_exit": "false", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_host": "127.0.0.1", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_port": "12201", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_new": "1000", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_recent": "10000", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stderr_prefix": "", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stop_at_utilization": "0.970000", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": "true", 2024-09-13T11:32:17.022 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_graylog": "false", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_journald": "false", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_stderr": "false", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_syslog": "false", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "max_rotating_auth_attempts": "10", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_action_on_write_error": "1", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_alternate_name_max": "8192", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_asio_thread_count": "2", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_export_pin": "true", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_dirs": "true", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_interval": "5", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-13T11:32:17.023 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_interval": "10", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max": "-1", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max_until": "-1", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_merge_size": "50", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_midchunk": "0.300000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_start": "0.200000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_minchunk": "0.001000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_mode": "0", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_max": "1.200000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_min": "0.800000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_overload_epochs": "2", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-13T11:32:17.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_bits": "3", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_size": "10000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_target_decay": "10.000000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_grace": "15.000000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_interval": "4.000000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_mid": "0.700000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_delay": "0", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-13T11:32:17.025 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_release_free_interval": "10", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_reservation": "0.050000", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_interval": "1", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_threshold": "262144", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_prealloc_inos": "1000", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_connect_bootstrapping": "false", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_damage_table_max_entries": "10000", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-13T11:32:17.026 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_auth_pins": "false", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_frag": "false", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_scatterstat": "false", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_subtrees": "false", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_decay_halflife": "5.000000", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_default_dir_hash": "2", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_defer_session_stale": "true", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_deny_all_reconnect": "false", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_keys_per_op": "16384", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_commit_size": "10", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_entries": "0", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_prefetch": "true", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-13T11:32:17.027 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_on_map": "false", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_early_reply": "true", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enable_op_tracker": "true", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enforce_unique_name": "true", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random": "true", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-13T11:32:17.028 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_summarize_threshold": "10", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_health_dummy": "false", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_migrator_session_race": "false", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_join_fs": "", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_journal_format": "1", 2024-09-13T11:32:17.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_create_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_dirfrag_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_export_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_import_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_expire_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_replay_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_link_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_mdstable_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_openc_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_rename_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_shutdown_at": "0", 2024-09-13T11:32:17.030 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_event_large_threshold": "524288", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_events_per_segment": "1024", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_events": "-1", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_segments": "128", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_pause": "false", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_segment_size": "0", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_threshold": "128", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-13T11:32:17.031 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_warn_factor": "2.000000", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_caps_per_client": "1048576", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_flushes": "100000", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_requests": "100000", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_export_size": "20971520", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_file_recover": "32", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_files": "64", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops": "8192", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_snaps_per_dir": "100", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_metrics_update_interval": "2", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_per_client": "100", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_working_set": "10000", 2024-09-13T11:32:17.032 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_numa_node": "-1", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_complaint_time": "30.000000", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_duration": "600", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_size": "20", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_size": "20", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_log_threshold": "5", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_grace": "15", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_interval": "5", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_caps": "30000", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-13T11:32:17.033 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_threshold": "262144", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_interval": "1.000000", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_gid": "0", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_uid": "0", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-13T11:32:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_shutdown_check": "0", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_skip_ino": "0", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_max_uid": "4294967294", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_min_uid": "0", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_rstat": "false", 2024-09-13T11:32:17.035 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_standby_replay_damaged": "false", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_symlink_recovery": "true", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_exports": "0", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_fragments": "0", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_tick_interval": "5.000000", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_valgrind_exit": "false", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_backtrace": "1", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_scatter": "false", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_sessions": "false", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool_debug": "false", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_device_bytes": "1073741824", 2024-09-13T11:32:17.036 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_set": "false", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_size": "65536", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_bytes": "134217728", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_messages": "512", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_disabled_modules": "", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_enable_op_tracker": "true", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_max_pg_creating": "1024", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_max_pg_num_change": "128", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_bytes": "134217728", 2024-09-13T11:32:17.037 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_messages": "128", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_bytes": "134217728", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_messages": "128", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_op_history_duration": "600", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_op_history_size": "20", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_op_log_threshold": "5", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_bytes": "536870912", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_messages": "8192", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_pool": "true", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-13T11:32:17.038 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_standby_modules": "true", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_period": "5", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_threshold": "5", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_tick_period": "2", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_delete": "true", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_size_one": "true", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_auth_validate_all_caps": "true", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_bytes": "104857600", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_directed_command_retry": "5", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-13T11:32:17.039 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_parallel": "3", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_log_interval": "1.000000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_interval": "10.000000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_target_rank": "-1", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_level": "debug", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_file": "true", 2024-09-13T11:32:17.040 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_journald": "false", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_bootstrap": "false", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_start": "false", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_trim": "true", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cpu_threads": "4", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_crush_min_required_version": "hammer", 2024-09-13T11:32:17.041 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_daemon_bytes": "419430400", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_crit": "5", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_warn": "5", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_size_warn": "16106127360", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_json": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_transactions": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_extra_checks": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_reef": "false", 2024-09-13T11:32:17.042 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_squid": "false", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_down_mkfs_grace": "120", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_down_uptime_grace": "60", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_default_strategy": "3", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_timeout": "5.000000", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_divisor": "2", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_enable_op_tracker": "true", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_fake_pool_delete": "false", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_force_quorum_join": "false", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-13T11:32:17.043 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_globalid_prealloc": "10000", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_detail_to_clog": "false", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_log_update_period": "5", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_max_detail": "50", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog": "true", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_interval": "600", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host": "172.21.15.38,172.21.15.118,172.21.15.154", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host_override": "", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_initial_members": "", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-13T11:32:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease": "5.000000", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_full_interval": "50", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max": "10000", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max_summary": "50", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_epochs": "500", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_osd": "10000", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pg_per_osd": "10000", 2024-09-13T11:32:17.045 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pool_pg_num": "65536", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_force_trim_to": "0", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_skip_sanity": "false", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_autotune": "true", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_target": "2147483648", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_beacon_grace": "30", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_digest_period": "5", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_inactive_grace": "60", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_min_osdmap_epochs": "50", 2024-09-13T11:32:17.046 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_complaint_time": "30", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_duration": "600", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_size": "20", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_size": "20", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_log_threshold": "5", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_in": "false", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-13T11:32:17.047 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size": "500", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_interval": "600", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_force_trim_to": "0", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-13T11:32:17.048 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_down_reporters": "2", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_report_timeout": "900", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-13T11:32:17.049 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_stuck_threshold": "60", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_probe_timeout": "2.000000", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_change": "0.050000", 2024-09-13T11:32:17.050 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_osds": "4", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_interval": "300", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_max_keys": "100", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_timeout": "300", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_session_timeout": "300", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_smart_report_timeout": "5", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stat_smooth_intervals": "6", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_min_size": "2", 2024-09-13T11:32:17.051 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_size": "4", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_debug": "false", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_provider_kill_at": "0", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_requester_kill_at": "0", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_timeout": "60.000000", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_target_pg_per_osd": "100", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_tick_interval": "5", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_interval": "300.000000", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_use_min_delay_socket": "false", 2024-09-13T11:32:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_older_version_delay": "604800", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_misplaced": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_older_version": "true", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-13T11:32:17.053 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "monmap": "", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_tracing_enabled": "false", 2024-09-13T11:32:17.054 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_op_threads": "3", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_cm": "false", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_device_name": "", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_dscp": "96", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_local_gid": "", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_port_num": "1", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_sl": "3", 2024-09-13T11:32:17.055 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_support_srq": "true", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_type": "ib", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_reap_threshold": "5", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_before_connect": "false", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv4": "true", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv6": "false", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr1": "false", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr2": "true", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_max": "7568", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_min": "6800", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_count": "3", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_delay": "5", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_client": "false", 2024-09-13T11:32:17.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mds": "false", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mgr": "false", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mon": "false", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_osd": "false", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_client_mode": "crc secure", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_mode": "crc secure", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_type": "", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_compress_secure": "false", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_idle_timeout": "900", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_ready_timeout": "10", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_data": "true", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_header": "true", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bad_msg": "false", 2024-09-13T11:32:17.057 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bug": "true", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_old_message": "true", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_skipped_message": "false", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_coremask": "0xF", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_enable_tso": "true", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hugepages": "", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-13T11:32:17.058 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_lro": "true", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_memory_channel": "4", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_pmd": "", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_port_id": "0", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_on_send": "false", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_initial_backoff": "0.200000", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_max": "1.000000", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_type": "", 2024-09-13T11:32:17.059 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_network_congestion": "0", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_socket_failures": "5000", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_learn_addr_from_peer": "true", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_accept_failures": "4", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_backoff": "15.000000", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_client_mode": "secure crc", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_service_mode": "secure crc", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_min_size": "1024", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_mode": "none", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-13T11:32:17.060 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_min_cost": "65536", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_public_type": "", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_service_mode": "crc secure", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_listen_backlog": "512", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_nodelay": "true", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_rcvbuf": "0", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_type": "async", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "no_config_file": "false", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_completion_locks_per_session": "32", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-13T11:32:17.061 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_ops": "1024", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_tick_interval": "5.000000", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_timeout": "10.000000", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_blackhole": "false", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "openssl_engine_opts": "", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_delay_time": "5.000000", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_hist_halflife": "1000", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_low_ops": "2", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_ops": "4", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-13T11:32:17.062 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_slop": "0.020000", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_asio_thread_count": "2", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_async_recovery_min_cost": "100", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_max": "512", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_min": "64", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_degraded": "false", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_peering": "false", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_unfound": "true", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_beacon_report_interval": "300", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_duration": "30", 2024-09-13T11:32:17.063 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_max_block_size": "67108864", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blkin_trace_all": "false", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_for_log_corruption": "false", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-13T11:32:17.064 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_update_on_start": "true", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_cap": "256", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_size_cap": "524288000", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_op_priority": "63", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_watch_timeout": "30", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_command_max_records": "256", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_compact_on_start": "false", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_on_start": "true", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_weight_set": "true", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-13T11:32:17.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_feed_pullee": "-1", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_misdirected_ops": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_acting_change": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_purge_strays": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_op_order": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-13T11:32:17.066 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_shutdown": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_trim_objects": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_snaps": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_keys": "1024", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-13T11:32:17.067 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_stride": "524288", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_notify_timeout": "30", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep": "0.000000", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_discard_disconnected_ops": "true", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_ec_partial_reads": "true", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_enable_op_tracker": "true", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-13T11:32:17.068 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_info": "true", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown": "true", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_function_tracing": "false", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_grace": "20", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_interval": "6", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_peers": "10", 2024-09-13T11:32:17.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_size": "2000", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_stale": "600", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_max_size": "100000", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_min_size": "1000", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_size": "5120", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_kill_backfill_at": "0", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-13T11:32:17.070 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_cache_size": "50", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_dedup": "true", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max": "40", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_share_max_epochs": "40", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_name_len": "100", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_size": "0", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_backfills": "1", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_count": "5", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_period": "600", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_name_len": "2048", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_namespace_len": "256", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_size": "134217728", 2024-09-13T11:32:17.071 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_blocked_by": "16", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_log_entries": "2", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pgls": "1024", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_cost": "8388608", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_objects": "10", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_scrubs": "3", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_trimming_pgs": "2", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_op_reply_len": "64", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_size": "90", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-13T11:32:17.072 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_profile": "balanced", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-13T11:32:17.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_base": "805306368", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_min": "134217728", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target": "4294967296", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_autotune": "false", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_pg_log_entries": "1", 2024-09-13T11:32:17.074 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_recovery_priority": "0", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_interval": "5", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_cache_shards": "32", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_op_tracker_shard": "32", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_auto_affinity": "true", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_node": "-1", 2024-09-13T11:32:17.075 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_prefer_iface": "true", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objecter_finishers": "1", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore": "bluestore", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_fuse": "false", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_tracing": "false", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_complaint_time": "30.000000", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_duration": "600", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_size": "20", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_size": "20", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_log_threshold": "5", 2024-09-13T11:32:17.076 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards": "0", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_hdd": "5", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_ssd": "8", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_min_cost": "65536", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue_cut_off": "high", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_timeout": "15", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_open_classes_on_start": "true", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_os_flags": "0", 2024-09-13T11:32:17.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_peering_op_priority": "255", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_cost": "1048576", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_priority": "5", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_max": "10000", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_min": "0", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-13T11:32:17.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crimson": "false", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-13T11:32:17.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flags": "0", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_min_size": "2", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_num": "32", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pgp_num": "0", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_ratio": "70", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_size": "3", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_type": "replicated", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_push_per_object_cost": "1000", 2024-09-13T11:32:17.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap": "true", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_cost": "20971520", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active": "0", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_single_start": "1", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_priority": "3", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_priority": "5", 2024-09-13T11:32:17.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep": "0.000000", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_repair_during_recovery": "false", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_requested_scrub_priority": "120", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair": "false", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_hour": "0", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_week_day": "0", 2024-09-13T11:32:17.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_max": "25", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_min": "5", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_cost": "52428800", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_during_recovery": "false", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_hour": "0", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_week_day": "0", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_event_cost": "4096", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_invalid_stats": "true", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_preemptions": "5", 2024-09-13T11:32:17.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_priority": "5", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_sleep": "0.000000", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_skip_data_digest": "false", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_smart_report_timeout": "5", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_cost": "1048576", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_priority": "5", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-13T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_transaction_size": "30", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-13T11:32:17.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tracing": "false", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_use_stale_snap": "false", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "osdc_blkin_trace_all": "false", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_kill_at": "0", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_max_join_drift": "10", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min": "500", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min_wait": "0.050000", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_propose_interval": "1.000000", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max": "500", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_min": "10", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_stash_full_interval": "25", 2024-09-13T11:32:17.086 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_max": "500", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_min": "250", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "perf": "true", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "-", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrv": "", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "public_bind_addr": "-", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network": "", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network_interface": "", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_busy_polling": "false", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_enabled": "false", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_session_max_number": "256", 2024-09-13T11:32:17.087 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_mon_op_timeout": "0", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_osd_op_timeout": "0", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_tracing": "false", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_atime_update_interval": "60", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_parent_reads": "false", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_snap_reads": "false", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blkin_trace_all": "false", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache": "true", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-13T11:32:17.088 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_policy": "writearound", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_size": "33554432", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_clone_copy_on_read": "false", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_compression_hint": "none", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_concurrent_management_ops": "10", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_clone_format": "auto", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_data_pool": "", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_features": "61", 2024-09-13T11:32:17.089 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_format": "2", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_map_options": "", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_order": "22", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_pool": "rbd", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_count": "0", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_unit": "0", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_enable_alloc_hint": "true", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler": "simple", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-13T11:32:17.090 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_order": "24", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_pool": "", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_splay_width": "4", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_parent_reads": "false", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_snap_reads": "false", 2024-09-13T11:32:17.091 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-13T11:32:17.092 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-13T11:32:17.093 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mtime_update_interval": "60", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_non_blocking_aio": "true", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_thread_timeout": "60", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_threads": "1", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_parent_cache_enabled": "false", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_plugins": "", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst": "0", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-13T11:32:17.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_limit": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_exclude_ops": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_limit": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-13T11:32:17.095 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_read_from_replica_policy": "default", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_skip_partial_discard": "true", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_tracing": "false", 2024-09-13T11:32:17.096 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_names": "true", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_pool": "true", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_base_url": "", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_log_level": "", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_acl_grants_max_num": "100", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_admin_entry": "admin", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_asio_assert_yielding": "false", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_backend_store": "rados", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_barbican_url": "", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_beast_enable_async": "true", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache": "false", 2024-09-13T11:32:17.097 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_enabled": "true", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_expiry_interval": "900", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_lru_size": "25000", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_config_store": "rados", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_content_length_compat": "false", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress": "true", 2024-09-13T11:32:17.098 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cors_rules_max_num": "100", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cross_domain_policy": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_addr": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_password": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_username": "", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_require_ssl": "true", 2024-09-13T11:32:17.099 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-13T11:32:17.100 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_addr": "", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_auth": "token", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_namespace": "", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_prefix": "", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_token_file": "", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_buffersize": "524288", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-13T11:32:17.101 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_time": "300", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-13T11:32:17.102 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_changes_size": "1000", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_num_shards": "128", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_window": "30", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-13T11:32:17.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_disable_s3select": "false", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-13T11:32:17.104 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_name": "", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_s3website_name": "", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dynamic_resharding": "true", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_gc_threads": "true", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_lc_threads": "true", 2024-09-13T11:32:17.105 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_mdsearch": "true", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_ops_log": "false", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_quota_threads": "true", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_static_website": "false", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_usage_log": "false", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enforce_swift_acls": "true", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_exit_timeout_secs": "120", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_expose_bucket": "false", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_extended_http_attrs": "", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_filter": "none", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontends": "beast port=7480", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred": "50", 2024-09-13T11:32:17.106 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_objs": "32", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_period": "3600", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_init_timeout": "300", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-13T11:32:17.107 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_connection_idle": "300", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_domain": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password_path": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_project": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_tenant": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token_path": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_user": "", 2024-09-13T11:32:17.108 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_domain": "", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_password": "", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_project": "", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_user": "", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_url": "", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_debug_interval": "-1", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_lock_max_time": "90", 2024-09-13T11:32:17.109 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_objs": "32", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_rules": "1000", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_worker": "3", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_thread_delay": "0", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_dnattr": "uid", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchfilter": "", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_uri": "", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-13T11:32:17.110 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_http_headers": "", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name_utc": "false", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_name_len": "0", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_size": "0", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_chunk_size": "4194304", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-13T11:32:17.111 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_listing_results": "1000", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_notify_retries": "10", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_param_size": "1048576", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_size": "5368709120", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_slo_entries": "1000", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_log_max_shards": "64", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mp_lock_max_time": "600", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-13T11:32:17.112 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_max_gc": "300", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-13T11:32:17.113 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_async_rados_threads": "32", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_control_oids": "8", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_numa_node": "-1", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_chunk_size": "100", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_gc_interval": "600", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-13T11:32:17.114 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_timeout": "600", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_tracing": "false", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_token": "", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_url": "", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_verify_ssl": "true", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_rados": "false", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_socket_path": "", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-13T11:32:17.115 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval": "2.000000", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lanes": "3", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_partitions": "3", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_continue": "true", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-13T11:32:17.116 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_tracing": "false", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm": "", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_id": "", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region": "", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_request_uri": "", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_batch_size": "64", 2024-09-13T11:32:17.117 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_max_aio": "128", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_num_logs": "16", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_thread_interval": "600", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_resolve_cname": "false", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_run_sync_thread": "true", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-13T11:32:17.118 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_scheduler_type": "throttler", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_script_uri": "", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_service_provider_name": "", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_id": "", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_secret": "", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_entry": "sts", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_key": "sts", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_min_session_duration": "900", 2024-09-13T11:32:17.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_token_introspection_url": "", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_account_in_url": "false", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_entry": "auth", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_url": "", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_custom_header": "", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_need_stats": "true", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_tenant_name": "", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_token_expiration": "86400", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url": "", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url_prefix": "swift", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-13T11:32:17.120 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_lease_period": "120", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_thread_pool_size": "512", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-13T11:32:17.121 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_comment": "", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_createby": "", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_encoding": "", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_flag": "false", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_origin": "", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_tracker": "", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_trust_forwarded_https": "false", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-13T11:32:17.122 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_shards": "32", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_user_shards": "1", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_use_opa_authz": "false", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache": "false", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_max_buckets": "1000", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_policies_max_num": "100", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_unique_email": "true", 2024-09-13T11:32:17.123 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_verify_ssl": "true", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone": "", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_id": "", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup": "", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_id": "", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_auto_session_clear": "true", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_block_size": "4096", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-13T11:32:17.124 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_size": "536870912", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_index_type": "binary_search", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-13T11:32:17.125 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_partition_filters": "false", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_perf": "false", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "run_dir": "/var/run/ceph", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_block_create": "true", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cache_lru_size": "67108864", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_max_object_size": "16777216", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_device_size": "53687091200", 2024-09-13T11:32:17.126 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_full_integrity_check": "false", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_capacity": "16", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_main_device_type": "SSD", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_segment_size": "67108864", 2024-09-13T11:32:17.127 INFO:teuthology.orchestra.run.smithi038.stdout: "set_keepcaps": "false", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "setgroup": "", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser": "", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser_match_path": "", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "thp": "false", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_default_timeout": "60", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "throttler_perf_counter": "true", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tmp_dir": "/tmp", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "uadk_compressor_enabled": "false", 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-13T11:32:17.128 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:17.131 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-13T11:32:17.249 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-09-13T11:32:17.249 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:32:17.259 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-09-13T11:32:17.259 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:32:17.269 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-09-13T11:32:17.269 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:32:17.279 DEBUG:teuthology.orchestra.run.smithi154:> type ceph-objectstore-tool 2024-09-13T11:32:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:32:17.298 DEBUG:teuthology.orchestra.run.smithi038:> type ceph-objectstore-tool 2024-09-13T11:32:17.313 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:32:17.313 DEBUG:teuthology.orchestra.run.smithi204:> type ceph-objectstore-tool 2024-09-13T11:32:17.331 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:32:17.331 DEBUG:teuthology.orchestra.run.smithi118:> type ceph-objectstore-tool 2024-09-13T11:32:17.352 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:32:17.352 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T11:32:17.361 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T11:32:17.361 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:32:17.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create low_tier 4' 2024-09-13T11:32:17.362 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-13T11:32:17.362 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:17.363 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:32:17.363 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:32:17.363 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-13T11:32:17.363 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T11:32:17.363 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T11:32:17.363 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok config set filestore_inject_stall 3 2024-09-13T11:32:17.364 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-13T11:32:17.364 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-13T11:32:17.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:17.365 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-13T11:32:17.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:32:17.366 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-13T11:32:17.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:32:17.392 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:17.391+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:17.466 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:17.464+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:17.482 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:17.483 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:17.489 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:17.489 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T11:32:17.489 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:17.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:32:17.500 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:17.500 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:17.567 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:17.566+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:17.607 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:17.607 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:17.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:32:17.669 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:17.667+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:17.720 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:17.721 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:17.724 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:17.724 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:17.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:32:17.745 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-09-13T11:32:17.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:32:17.762 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:17.763 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:17.770 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:17.769+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:17.780 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:17.781 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:17.798 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:17.799 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:17.817 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:17.817 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:17.834 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:17.835 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:17.851 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:17.851 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:17.854 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:17.854 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:17.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:32:17.868 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:17.871 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:17.869+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:17.887 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:17.887 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:17.905 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:17.906 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:17.922 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:17.923 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:17.940 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:17.941 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:17.958 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:17.956+0000 7f1a71494640 0 -- 172.21.15.38:0/3738292286 >> v2:172.21.15.204:6812/501019151 conn(0x7f1a440334d0 msgr2=0x7f1a440358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-13T11:32:17.972 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:17.971+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:17.976 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:17.976 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:17.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:32:18.073 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:18.072+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:18.133 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:18.133 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:18.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:32:18.174 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:18.173+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:18.175 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:18.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:18.250 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:18.250 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:18.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:32:18.274 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:18.273+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:18.364 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:18.365 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:18.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:32:18.375 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:18.374+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:18.416 INFO:teuthology.orchestra.run.smithi204.stderr:pool 'low_tier' created 2024-09-13T11:32:18.418 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:32:18.430 INFO:teuthology.run_tasks:Running task rados... 2024-09-13T11:32:18.440 INFO:tasks.rados:Beginning rados... 2024-09-13T11:32:18.441 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'set_chunk': 30, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-09-13T11:32:18.441 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore odsync write': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 0.5}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T11:32:18.441 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'set_chunk': 30, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-09-13T11:32:18.444 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-13T11:32:18.452 INFO:tasks.rados:joining rados 2024-09-13T11:32:18.452 INFO:tasks.rados:clients are ['client.0'] 2024-09-13T11:32:18.452 INFO:tasks.rados:starting run 0 out of 1 2024-09-13T11:32:18.452 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-13T11:32:18.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-13T11:32:18.476 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:18.474+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:18.482 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:18.483 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:18.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:32:18.577 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:18.576+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:18.641 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:18.642 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:18.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:32:18.695 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:18.694+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:18.759 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:18.759 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:18.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:32:18.808 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:18.807+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:18.879 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:18.877+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:18.914 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:18.914 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:18.922 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:32:18.980 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:18.978+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:19.037 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.038 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.046 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:32:19.080 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:19.079+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:19.150 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.150 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:32:19.181 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:19.180+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:19.261 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.261 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:32:19.282 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:19.281+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:19.372 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.373 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:32:19.383 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:19.382+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:19.419 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_0' created 2024-09-13T11:32:19.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-13T11:32:19.484 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:19.482+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:19.486 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.486 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:32:19.585 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:19.583+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:19.597 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.598 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.606 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:32:19.686 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:19.685+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:19.709 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.709 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.718 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:32:19.787 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:19.786+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:19.823 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.824 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:32:19.887 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:19.886+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:19.935 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:19.936 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:19.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:32:19.988 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:19.987+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:20.089 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:20.089 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:20.089 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:20.087+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:20.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:32:20.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:20.190 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:20.188+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:20.238 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:20.238 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:20.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:32:20.290 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:20.289+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:20.318 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:20.318 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:20.335 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:20.336 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:20.353 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-09-13T11:32:20.354 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:32:20.371 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:20.371 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:20.387 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:20.388 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:20.388 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:20.388 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:20.391 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:20.390+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:20.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:32:20.403 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:20.404 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:20.421 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:20.422 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:20.422 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-13T11:32:20.443 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:20.444 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:20.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:32:20.460 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:20.460 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:20.477 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:20.477 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:20.492 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:20.491+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:20.494 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:20.495 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:20.512 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:20.513 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:20.517 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:20.517 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:20.517 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:20.517 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:20.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:32:20.529 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:20.529 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:20.546 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:20.546 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:20.564 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:20.564 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:20.565 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:20.564+0000 7f0d129dd640 0 -- 172.21.15.38:0/3522260195 >> v2:172.21.15.204:6812/501019151 conn(0x7f0d0c224d40 msgr2=0x7f0d0c227130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:20.583 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:20.583 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:20.592 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:20.591+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:20.633 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:20.633 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:20.633 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:20.633 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:20.634 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:20.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:32:20.693 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:20.693+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:20.744 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:20.744 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:20.745 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [] 2024-09-13T11:32:20.745 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:20.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:32:20.795 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:20.794+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:20.858 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:20.858 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:20.858 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:20.858 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:20.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:32:20.897 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:20.897+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:20.970 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:20.970 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:20.970 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:20.970 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:20.970 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:20.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:32:20.999 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:20.997+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:21.083 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:21.083 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:21.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:21.083 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [] 2024-09-13T11:32:21.083 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:21.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:32:21.100 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:21.099+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:21.194 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:21.194 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:21.194 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:21.195 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:21.200 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:21.199+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:21.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:32:21.305 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:21.300+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:21.307 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:21.307 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:21.307 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:21.307 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:21.307 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:21.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:32:21.402 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:21.401+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:21.427 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:32:21.440 DEBUG:teuthology.orchestra.run.smithi204:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0 2024-09-13T11:32:21.461 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:21.652 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:21.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:21.652 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:32:21.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:07.366068+0000", 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.094591851000001, 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012356240000000001, 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366068+0000", 2024-09-13T11:32:21.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366068+0000", 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366069+0000", 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.23e-07 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366080+0000", 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1267000000000001e-05 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366123+0000", 2024-09-13T11:32:21.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2765000000000001e-05 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366164+0000", 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1072999999999998e-05 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:21.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366243+0000", 2024-09-13T11:32:21.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8857999999999996e-05 2024-09-13T11:32:21.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:21.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.366269+0000", 2024-09-13T11:32:21.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5959999999999999e-05 2024-09-13T11:32:21.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.367258+0000", 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00098919000000000003 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:07.367304+0000", 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5688e-05 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:21.658 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:08.380374+0000", 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.080285421999999, 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011701019999999999, 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:21.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380374+0000", 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380374+0000", 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380375+0000", 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9399999999999995e-07 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380384+0000", 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2730000000000008e-06 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:21.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380409+0000", 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4805999999999999e-05 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380427+0000", 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7693000000000002e-05 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380570+0000", 2024-09-13T11:32:21.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014295000000000001 2024-09-13T11:32:21.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:21.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.380631+0000", 2024-09-13T11:32:21.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1774999999999997e-05 2024-09-13T11:32:21.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.381507+0000", 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00087542200000000005 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:08.381545+0000", 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7688999999999998e-05 2024-09-13T11:32:21.664 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:21.665 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:21.665 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:21.665 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:21.665 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:21.665 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:21.667 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight read -> 100 2024-09-13T11:32:21.667 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight write -> 25 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight delete -> 10 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight set_chunk -> 30 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight tier_promote -> 10 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight tier_evict -> 10 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight tier_flush -> 5 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:adding op weight write_excl -> 25 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:ceph version 19.3.0-4697-gd58f7306 (d58f73069a8157a9ace12a0247ee3e8497707ea9) squid (dev) 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout:Configuration: 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout: Number of operations: 1500 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout: Number of objects: 50 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout: Max in flight operations: 16 2024-09-13T11:32:21.668 INFO:tasks.rados.rados.0.smithi204.stdout: Object size (in bytes): 4000000 2024-09-13T11:32:21.669 INFO:tasks.rados.rados.0.smithi204.stdout: Write stride min: 400000 2024-09-13T11:32:21.669 INFO:tasks.rados.rados.0.smithi204.stdout: Write stride max: 800000 2024-09-13T11:32:21.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:32:21.670 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:21.666+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:21.768 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:21.766+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:21.775 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:21.775 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:21.775 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:21.775 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:21.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:32:21.870 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:21.869+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:21.929 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:21.929 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:21.929 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:21.929 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:21.929 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:21.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:32:21.970 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:21.969+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:22.043 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:22.043 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:22.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:22.044 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [] 2024-09-13T11:32:22.044 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:22.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:32:22.097 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:22.096+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:22.168 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:22.168 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:22.168 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:22.168 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:22.171 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:22.170+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:22.177 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:32:22.271 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:22.271+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:22.321 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:22.321 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:22.321 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:22.321 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:22.322 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:22.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:32:22.373 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:22.372+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:22.437 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:22.437 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:89 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'63, mlcod=24'60)", 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:04.981303+0000", 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.453779215000001, 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0062066819999999998, 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981303+0000", 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981303+0000", 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981304+0000", 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1399999999999997e-07 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981309+0000", 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.006e-06 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981317+0000", 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8619999999999992e-06 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981323+0000", 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5559999999999998e-06 2024-09-13T11:32:22.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981359+0000", 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6355e-05 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981372+0000", 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2503999999999999e-05 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.987484+0000", 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006111839 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.987510+0000", 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6046000000000001e-05 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:90 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'64, mlcod=24'60)", 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:04.981788+0000", 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.453293782999999, 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006234346, 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981788+0000", 2024-09-13T11:32:22.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981788+0000", 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981789+0000", 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9999999999999998e-07 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981854+0000", 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981863+0000", 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6589999999999996e-06 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.981869+0000", 2024-09-13T11:32:22.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4709999999999998e-06 2024-09-13T11:32:22.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983676+0000", 2024-09-13T11:32:22.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001807468 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983691+0000", 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4831000000000001e-05 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.987999+0000", 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043078220000000002 2024-09-13T11:32:22.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.988023+0000", 2024-09-13T11:32:22.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4181000000000001e-05 2024-09-13T11:32:22.448 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:92 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'66, mlcod=24'61)", 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:04.982884+0000", 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.452198715000002, 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060305910000000001, 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.449 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.982884+0000", 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.982884+0000", 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.982884+0000", 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.982888+0000", 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.982895+0000", 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2450000000000001e-06 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.982900+0000", 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.237e-06 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984587+0000", 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016866229999999999 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984599+0000", 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2129e-05 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.988888+0000", 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042883779999999998 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.988914+0000", 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6605999999999999e-05 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:93 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'67, mlcod=24'61)", 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:04.983480+0000", 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.451601901, 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075470540000000001, 2024-09-13T11:32:22.455 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.456 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.456 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.456 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983480+0000", 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983480+0000", 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983481+0000", 2024-09-13T11:32:22.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6899999999999996e-07 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983517+0000", 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5389000000000003e-05 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983527+0000", 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0293e-05 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983534+0000", 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984827+0000", 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0012929829999999999 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984838+0000", 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1565e-05 2024-09-13T11:32:22.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.990993+0000", 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061549339999999999 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.991027+0000", 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4409999999999998e-05 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:94 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'68, mlcod=24'61)", 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:04.983934+0000", 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.451148286999999, 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0071125390000000002, 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983934+0000", 2024-09-13T11:32:22.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983934+0000", 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983935+0000", 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8899999999999999e-07 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983939+0000", 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7679999999999999e-06 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983948+0000", 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1610000000000003e-06 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.983954+0000", 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1210000000000003e-06 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.987547+0000", 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035937130000000001 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.987560+0000", 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2559e-05 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.991031+0000", 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034711899999999999 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.991047+0000", 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5438e-05 2024-09-13T11:32:22.465 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:95 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'69, mlcod=24'61)", 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:04.984387+0000", 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.450694982999998, 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066783559999999999, 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.466 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984387+0000", 2024-09-13T11:32:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984387+0000", 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984388+0000", 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4199999999999995e-07 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984393+0000", 2024-09-13T11:32:22.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0579999999999997e-06 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984401+0000", 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2339999999999994e-06 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984407+0000", 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8769999999999998e-06 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.987785+0000", 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033781290000000001 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.987798+0000", 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.31e-05 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.991048+0000", 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032491220000000001 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.991066+0000", 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8094e-05 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:96 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'70, mlcod=24'61)", 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:04.984938+0000", 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.450143986, 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061432359999999998, 2024-09-13T11:32:22.471 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984938+0000", 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984938+0000", 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.984947+0000", 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7930000000000003e-06 2024-09-13T11:32:22.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.985017+0000", 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9624999999999998e-05 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.985026+0000", 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4299999999999995e-06 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.985031+0000", 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9189999999999998e-06 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.988081+0000", 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030501920000000002 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.988094+0000", 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2785999999999999e-05 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.991066+0000", 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029724410000000001 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:04.991082+0000", 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.505e-05 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:114 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'84, mlcod=24'80)", 2024-09-13T11:32:22.476 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.070050+0000", 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.365032187000001, 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059660970000000001, 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070050+0000", 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070050+0000", 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070051+0000", 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4499999999999997e-07 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070136+0000", 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4983999999999998e-05 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070145+0000", 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0010000000000007e-06 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070154+0000", 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070192+0000", 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8303000000000002e-05 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070206+0000", 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3883000000000001e-05 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075982+0000", 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0057757590000000001 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076016+0000", 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4671000000000001e-05 2024-09-13T11:32:22.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:115 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'85, mlcod=24'80)", 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.070497+0000", 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.364585542, 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010713258, 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070497+0000", 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070497+0000", 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070497+0000", 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1100000000000001e-07 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070501+0000", 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.693e-06 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070508+0000", 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.070512+0000", 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9999999999999998e-06 2024-09-13T11:32:22.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072944+0000", 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024329249999999998 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072957+0000", 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2353e-05 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081184+0000", 2024-09-13T11:32:22.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0082276329999999998 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081210+0000", 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5534999999999999e-05 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:116 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'86, mlcod=24'81)", 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.071049+0000", 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.364033085999999, 2024-09-13T11:32:22.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010185148999999999, 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071049+0000", 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071049+0000", 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071050+0000", 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2100000000000002e-07 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071060+0000", 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0695e-05 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071067+0000", 2024-09-13T11:32:22.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.224e-06 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071076+0000", 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0010000000000007e-06 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073174+0000", 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020983519999999999 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073189+0000", 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5217e-05 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081213+0000", 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0080238150000000001 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081234+0000", 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1424e-05 2024-09-13T11:32:22.489 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'87, mlcod=24'81)", 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.363538731999999, 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0126724, 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-13T11:32:22.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071634+0000", 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9599999999999996e-05 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071640+0000", 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4980000000000003e-06 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071644+0000", 2024-09-13T11:32:22.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2919999999999997e-06 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073407+0000", 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001762149 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073417+0000", 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0859999999999999e-05 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084190+0000", 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010772903 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084216+0000", 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5622e-05 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:118 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'88, mlcod=24'81)", 2024-09-13T11:32:22.494 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.363018717999999, 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012784627999999999, 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4900000000000001e-07 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072067+0000", 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2260000000000001e-06 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072074+0000", 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2820000000000002e-06 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072077+0000", 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5899999999999999e-06 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075773+0000", 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003695884 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075786+0000", 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2627e-05 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084816+0000", 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090302000000000004 2024-09-13T11:32:22.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084848+0000", 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2369999999999997e-05 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:119 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'89, mlcod=24'81)", 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.362568665000001, 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012362191, 2024-09-13T11:32:22.499 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072521+0000", 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6540000000000002e-06 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072531+0000", 2024-09-13T11:32:22.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8309999999999996e-06 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072536+0000", 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3090000000000003e-06 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076045+0000", 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035092769999999999 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076058+0000", 2024-09-13T11:32:22.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2992e-05 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084850+0000", 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087915289999999993 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084876+0000", 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5947000000000001e-05 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:120 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'90, mlcod=24'81)", 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072941+0000", 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.362140984, 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011957627, 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072941+0000", 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072941+0000", 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072942+0000", 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.32e-07 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073044+0000", 2024-09-13T11:32:22.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010217099999999999 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073054+0000", 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7899999999999994e-06 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073058+0000", 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076266+0000", 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003207938 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076278+0000", 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1464000000000001e-05 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084877+0000", 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085992389999999998 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084899+0000", 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1798e-05 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:121 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'91, mlcod=24'81)", 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.073396+0000", 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.361685827999999, 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011525894, 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.508 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073396+0000", 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073396+0000", 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073397+0000", 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.68e-07 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073404+0000", 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5599999999999999e-06 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073410+0000", 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7900000000000002e-06 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073415+0000", 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5920000000000002e-06 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081268+0000", 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078531829999999997 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081280+0000", 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2065999999999999e-05 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084901+0000", 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036210040000000001 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084922+0000", 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1131000000000001e-05 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.512 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:122 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'92, mlcod=24'82)", 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.361065971999999, 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013166437, 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074017+0000", 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9500000000000003e-07 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074022+0000", 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0529999999999997e-06 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074029+0000", 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8419999999999999e-06 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074032+0000", 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8060000000000001e-06 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081498+0000", 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074659669999999996 2024-09-13T11:32:22.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081510+0000", 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1265e-05 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087152+0000", 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056421889999999997 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087183+0000", 2024-09-13T11:32:22.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.082e-05 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'82)", 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.360553879000001, 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012918696, 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.517 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074529+0000", 2024-09-13T11:32:22.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8400000000000004e-07 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074601+0000", 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2106000000000004e-05 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074611+0000", 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0054000000000001e-05 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074616+0000", 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6349999999999997e-06 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084250+0000", 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096340109999999996 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084263+0000", 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3292000000000001e-05 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087416+0000", 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003152672 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087447+0000", 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1241999999999997e-05 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.521 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'82)", 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 17.360028687, 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012417658, 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5600000000000001e-07 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075058+0000", 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6689999999999999e-06 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075065+0000", 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9269999999999999e-06 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075069+0000", 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.963e-06 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084645+0000", 2024-09-13T11:32:22.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095762209999999993 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084657+0000", 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2017e-05 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087449+0000", 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002792189 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087471+0000", 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2215999999999999e-05 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 16.080770225999999, 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:32:22.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:32:22.530 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:08.374924+0000", 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.060158384999999, 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068177680000000001, 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:32:22.531 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 7 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.374924+0000", 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.374924+0000", 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.374926+0000", 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1430000000000001e-06 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.374940+0000", 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3879999999999999e-05 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.374969+0000", 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9309e-05 2024-09-13T11:32:22.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.374984+0000", 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4337e-05 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.375129+0000", 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000145518 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.375147+0000", 2024-09-13T11:32:22.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7632999999999999e-05 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.377090+0000", 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019433479999999999 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.377152+0000", 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1886999999999999e-05 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,10", 2024-09-13T11:32:22.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.377242+0000", 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9708000000000002e-05 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.381004+0000", 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037623359999999998 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.381390+0000", 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038634100000000001 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.381666+0000", 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027555800000000002 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.381703+0000", 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6922000000000003e-05 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:08.381742+0000", 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8847999999999998e-05 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.537 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.538 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.538 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:22.538 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:22.538 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:22.538 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:22.538 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:32:22.538 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:32:22.539 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-09-13T11:32:22.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-full-ratio .001 2024-09-13T11:32:22.540 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:32:22.541 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:22.538+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:22.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:22.640 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:22.638+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:22.686 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:22.686 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:22.686 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:22.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:32:22.733 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:22.733 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:22.741 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:22.739+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:22.747 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:22.748 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:22.765 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-09-13T11:32:22.765 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:32:22.782 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:22.783 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:22.798 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:22.799 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:22.815 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:22.816 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:22.833 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:22.833 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:22.837 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:22.837 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:22.837 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:22.837 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:22.838 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:22.841 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:22.840+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:22.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:32:22.849 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:22.850 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:22.866 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:22.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:22.883 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:22.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:22.901 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:22.901 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:22.918 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:22.935 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:22.935 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:22.942 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:22.940+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:22.950 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:22.950 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:22.952 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:22.952 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:22.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:32:22.969 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:22.970 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:22.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:22.969+0000 7f1e492b4640 0 -- 172.21.15.38:0/1380234324 >> v2:172.21.15.204:6812/501019151 conn(0x7f1e44224f80 msgr2=0x7f1e44227370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:23.042 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:23.041+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:23.063 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.063 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:32:23.158 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:23.156+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:23.190 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:23.190 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:23.218 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.218 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.227 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:32:23.244 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:23.243+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:23.345 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:23.344+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:23.370 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.370 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:32:23.435 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:32:23.434+0000 7f8355539640 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-09-13T11:32:23.440 INFO:teuthology.orchestra.run.smithi038.stderr:osd set-full-ratio 0.001 2024-09-13T11:32:23.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:32:23.446 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:23.445+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:23.523 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.523 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.531 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:32:23.550 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:23.549+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:23.635 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.635 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.643 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:32:23.651 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:23.650+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:23.747 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.747 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.754 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:23.753+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:23.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:32:23.854 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:23.854+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:23.864 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.864 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:32:23.957 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:23.956+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:23.982 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:23.982 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:23.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:32:24.058 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:24.057+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:24.138 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:24.138 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:24.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:32:24.191 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:24.190+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:24.254 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:24.254 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:24.260 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:24.259+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:24.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:32:24.361 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:24.360+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:24.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:32:24.441 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:32:24.458 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-09-13T11:32:24.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-full-ratio .95 2024-09-13T11:32:24.498 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:24.497+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:24.530 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:24.530 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:24.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:32:24.581 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:24.580+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:24.663 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:24.662+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:24.694 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:24.694 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:24.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:32:24.764 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:24.763+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:24.811 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:24.811 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:24.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:32:24.866 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:24.865+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:24.929 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:24.929 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:24.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:32:24.979 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:24.978+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:25.067 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:25.066+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:25.085 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:25.085 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:25.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:32:25.168 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:25.167+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:25.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:25.203 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:25.203 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:25.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:32:25.268 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:25.267+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:25.328 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:25.329 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:25.347 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:25.347 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:25.361 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:25.361 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:25.364 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-09-13T11:32:25.364 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:32:25.370 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:32:25.370 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:25.368+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:25.382 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:25.382 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:25.398 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:25.398 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:25.415 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:25.415 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:25.432 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:25.433 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:25.443 INFO:teuthology.orchestra.run.smithi038.stderr:osd set-full-ratio 0.95 2024-09-13T11:32:25.450 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:25.450 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:25.467 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:25.468 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:25.470 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:25.469+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:25.484 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:25.485 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:25.488 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:25.488 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:25.488 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:25.489 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:25.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:32:25.502 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:25.502 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:25.520 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:25.536 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:25.536 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:25.570 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:25.570+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:25.601 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:25.601 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:25.601 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:25.601 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:25.601 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:25.609 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:32:25.671 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:25.670+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:25.711 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:25.711 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:25.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:25.711 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [] 2024-09-13T11:32:25.711 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:25.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:32:25.754 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:25.754 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:25.771 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:25.772 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:25.772 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:25.771+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:25.773 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:25.772+0000 7fcc7676e640 0 -- 172.21.15.38:0/2630226511 >> v2:172.21.15.204:6812/501019151 conn(0x7fcc70225000 msgr2=0x7fcc702273f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:25.790 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:25.790 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:25.823 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:25.823 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:25.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:32:25.875 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:25.874+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:25.934 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:25.934 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:25.934 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:25.934 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:25.934 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:25.943 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:32:25.974 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:25.973+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:26.075 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:26.074+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.741112439999998, 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079674329999999995, 2024-09-13T11:32:26.085 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4344", 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343169+0000", 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343177+0000", 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4819999999999994e-06 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343189+0000", 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2306999999999999e-05 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343199+0000", 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343279+0000", 2024-09-13T11:32:26.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0025999999999996e-05 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343289+0000", 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0518e-05 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348428+0000", 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051380779999999999 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348510+0000", 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2226999999999997e-05 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348598+0000", 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8722999999999997e-05 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349014+0000", 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041537900000000001 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349279+0000", 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026517700000000001 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351081+0000", 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001802292 2024-09-13T11:32:26.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351110+0000", 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8651000000000001e-05 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351132+0000", 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069000000000002e-05 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.093 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.4344.0:1 2.3 e26/25 ondisk, result = 0)", 2024-09-13T11:32:26.093 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.349201+0000", 2024-09-13T11:32:26.093 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.735076024000001, 2024-09-13T11:32:26.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6221e-05, 2024-09-13T11:32:26.093 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.094 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.094 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:32:26.094 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349201+0000", 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349201+0000", 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349202+0000", 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8400000000000004e-07 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349205+0000", 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6500000000000002e-06 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349215+0000", 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2269999999999999e-06 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349220+0000", 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5380000000000002e-06 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349264+0000", 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3699999999999998e-05 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349272+0000", 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9400000000000002e-06 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349287+0000", 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5582000000000001e-05 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.4344.0:1 2.3 e26/25 ondisk, result = 0)", 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.351017+0000", 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.733259953000001, 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120086, 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.098 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351017+0000", 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351017+0000", 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351018+0000", 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7300000000000001e-07 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351030+0000", 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2398999999999999e-05 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351038+0000", 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:32:26.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351044+0000", 2024-09-13T11:32:26.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9460000000000003e-06 2024-09-13T11:32:26.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351074+0000", 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0233e-05 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351078+0000", 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.296e-06 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351137+0000", 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8733000000000002e-05 2024-09-13T11:32:26.102 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4344.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:07.366041+0000", 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.718235908, 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010717439999999999, 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366041+0000", 2024-09-13T11:32:26.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366041+0000", 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366042+0000", 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2600000000000002e-07 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366050+0000", 2024-09-13T11:32:26.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7970000000000006e-06 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366080+0000", 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.904e-05 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366097+0000", 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7169000000000001e-05 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366242+0000", 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014561700000000001 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.366281+0000", 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8346000000000002e-05 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.367062+0000", 2024-09-13T11:32:26.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000781702 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:07.367113+0000", 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0547000000000002e-05 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4344.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:08.380330+0000", 2024-09-13T11:32:26.108 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.703946648999999, 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00095151399999999998, 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380330+0000", 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380330+0000", 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380331+0000", 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4000000000000002e-07 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380339+0000", 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1559999999999995e-06 2024-09-13T11:32:26.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380367+0000", 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.76e-05 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380408+0000", 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1403999999999999e-05 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380475+0000", 2024-09-13T11:32:26.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6628e-05 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.380494+0000", 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9609000000000001e-05 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.381235+0000", 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074053699999999999 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:08.381282+0000", 2024-09-13T11:32:26.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7039999999999997e-05 2024-09-13T11:32:26.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.113 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:26.114 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:32:26.176 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:26.175+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:26.217 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:26.217 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:26.217 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:26.217 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:26.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:32:26.309 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:26.308+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:26.368 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:26.368 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:26.369 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:26.369 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:26.369 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:26.377 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:32:26.378 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:26.377+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:26.448 INFO:tasks.rados.rados.0.smithi204.stdout:1: write initial oid 1 2024-09-13T11:32:26.448 INFO:tasks.rados.rados.0.smithi204.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-13T11:32:26.453 INFO:tasks.rados.rados.0.smithi204.stdout:1: writing smithi20438004-1 from 416807 to 892056 tid 1 2024-09-13T11:32:26.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:32:26.459 INFO:tasks.rados.rados.0.smithi204.stdout:1: writing smithi20438004-1 from 1542129 to 2016807 tid 2 2024-09-13T11:32:26.459 INFO:tasks.rados.rados.0.smithi204.stdout:2: write initial oid 2 2024-09-13T11:32:26.460 INFO:tasks.rados.rados.0.smithi204.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-13T11:32:26.464 INFO:tasks.rados.rados.0.smithi204.stdout:2: writing smithi20438004-2 from 416807 to 892056 tid 1 2024-09-13T11:32:26.471 INFO:tasks.rados.rados.0.smithi204.stdout:2: writing smithi20438004-2 from 1542129 to 2016807 tid 2 2024-09-13T11:32:26.471 INFO:tasks.rados.rados.0.smithi204.stdout:3: write initial oid 3 2024-09-13T11:32:26.471 INFO:tasks.rados.rados.0.smithi204.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-13T11:32:26.476 INFO:tasks.rados.rados.0.smithi204.stdout:3: writing smithi20438004-3 from 433614 to 984112 tid 1 2024-09-13T11:32:26.482 INFO:tasks.rados.rados.0.smithi204.stdout:3: writing smithi20438004-3 from 1600611 to 2033614 tid 2 2024-09-13T11:32:26.482 INFO:tasks.rados.rados.0.smithi204.stdout:4: write initial oid 4 2024-09-13T11:32:26.482 INFO:tasks.rados.rados.0.smithi204.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-13T11:32:26.488 INFO:tasks.rados.rados.0.smithi204.stdout:4: writing smithi20438004-4 from 450421 to 1076168 tid 1 2024-09-13T11:32:26.494 INFO:tasks.rados.rados.0.smithi204.stdout:4: writing smithi20438004-4 from 1659093 to 2050421 tid 2 2024-09-13T11:32:26.494 INFO:tasks.rados.rados.0.smithi204.stdout:5: write initial oid 5 2024-09-13T11:32:26.494 INFO:tasks.rados.rados.0.smithi204.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-13T11:32:26.500 INFO:tasks.rados.rados.0.smithi204.stdout:5: writing smithi20438004-5 from 467228 to 1168224 tid 1 2024-09-13T11:32:26.502 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:26.501+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:26.505 INFO:tasks.rados.rados.0.smithi204.stdout:5: writing smithi20438004-5 from 1717575 to 2067228 tid 2 2024-09-13T11:32:26.505 INFO:tasks.rados.rados.0.smithi204.stdout:6: write initial oid 6 2024-09-13T11:32:26.505 INFO:tasks.rados.rados.0.smithi204.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-13T11:32:26.512 INFO:tasks.rados.rados.0.smithi204.stdout:6: writing smithi20438004-6 from 484035 to 1260280 tid 1 2024-09-13T11:32:26.517 INFO:tasks.rados.rados.0.smithi204.stdout:6: writing smithi20438004-6 from 2059704 to 2084035 tid 2 2024-09-13T11:32:26.517 INFO:tasks.rados.rados.0.smithi204.stdout:7: write initial oid 7 2024-09-13T11:32:26.517 INFO:tasks.rados.rados.0.smithi204.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-13T11:32:26.522 INFO:tasks.rados.rados.0.smithi204.stdout:7: writing smithi20438004-7 from 500842 to 952336 tid 1 2024-09-13T11:32:26.524 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:26.524 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:26.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:26.524 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [] 2024-09-13T11:32:26.524 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:26.528 INFO:tasks.rados.rados.0.smithi204.stdout:7: writing smithi20438004-7 from 1718186 to 2100842 tid 2 2024-09-13T11:32:26.528 INFO:tasks.rados.rados.0.smithi204.stdout:8: write initial oid 8 2024-09-13T11:32:26.529 INFO:tasks.rados.rados.0.smithi204.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-13T11:32:26.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:32:26.534 INFO:tasks.rados.rados.0.smithi204.stdout:8: writing smithi20438004-8 from 517649 to 1044392 tid 1 2024-09-13T11:32:26.540 INFO:tasks.rados.rados.0.smithi204.stdout:8: writing smithi20438004-8 from 1776668 to 2117649 tid 2 2024-09-13T11:32:26.540 INFO:tasks.rados.rados.0.smithi204.stdout:9: write initial oid 9 2024-09-13T11:32:26.540 INFO:tasks.rados.rados.0.smithi204.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-13T11:32:26.547 INFO:tasks.rados.rados.0.smithi204.stdout:9: writing smithi20438004-9 from 534456 to 1252801 tid 1 2024-09-13T11:32:26.552 INFO:tasks.rados.rados.0.smithi204.stdout:9: writing smithi20438004-9 from 1951503 to 2134456 tid 2 2024-09-13T11:32:26.552 INFO:tasks.rados.rados.0.smithi204.stdout:10: write initial oid 10 2024-09-13T11:32:26.552 INFO:tasks.rados.rados.0.smithi204.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-13T11:32:26.559 INFO:tasks.rados.rados.0.smithi204.stdout:10: writing smithi20438004-10 from 551263 to 1344857 tid 1 2024-09-13T11:32:26.564 INFO:tasks.rados.rados.0.smithi204.stdout:10: writing smithi20438004-10 from 1893632 to 2151263 tid 2 2024-09-13T11:32:26.564 INFO:tasks.rados.rados.0.smithi204.stdout:11: write initial oid 11 2024-09-13T11:32:26.564 INFO:tasks.rados.rados.0.smithi204.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-13T11:32:26.570 INFO:tasks.rados.rados.0.smithi204.stdout:11: writing smithi20438004-11 from 568070 to 1036913 tid 1 2024-09-13T11:32:26.576 INFO:tasks.rados.rados.0.smithi204.stdout:11: writing smithi20438004-11 from 1552114 to 2168070 tid 2 2024-09-13T11:32:26.576 INFO:tasks.rados.rados.0.smithi204.stdout:12: write initial oid 12 2024-09-13T11:32:26.576 INFO:tasks.rados.rados.0.smithi204.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-13T11:32:26.580 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:26.579+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:26.582 INFO:tasks.rados.rados.0.smithi204.stdout:12: writing smithi20438004-12 from 584877 to 1128969 tid 1 2024-09-13T11:32:26.588 INFO:tasks.rados.rados.0.smithi204.stdout:12: writing smithi20438004-12 from 1610596 to 2172599 tid 2 2024-09-13T11:32:26.589 INFO:tasks.rados.rados.0.smithi204.stdout:13: write initial oid 13 2024-09-13T11:32:26.589 INFO:tasks.rados.rados.0.smithi204.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-13T11:32:26.595 INFO:tasks.rados.rados.0.smithi204.stdout:13: writing smithi20438004-13 from 601684 to 1221025 tid 1 2024-09-13T11:32:26.600 INFO:tasks.rados.rados.0.smithi204.stdout:13: writing smithi20438004-13 from 1669078 to 2201684 tid 2 2024-09-13T11:32:26.601 INFO:tasks.rados.rados.0.smithi204.stdout:14: write initial oid 14 2024-09-13T11:32:26.601 INFO:tasks.rados.rados.0.smithi204.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-13T11:32:26.607 INFO:tasks.rados.rados.0.smithi204.stdout:14: writing smithi20438004-14 from 618491 to 1313081 tid 1 2024-09-13T11:32:26.613 INFO:tasks.rados.rados.0.smithi204.stdout:14: writing smithi20438004-14 from 2011207 to 2218491 tid 2 2024-09-13T11:32:26.613 INFO:tasks.rados.rados.0.smithi204.stdout:15: write initial oid 15 2024-09-13T11:32:26.613 INFO:tasks.rados.rados.0.smithi204.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-13T11:32:26.621 INFO:tasks.rados.rados.0.smithi204.stdout:15: writing smithi20438004-15 from 635298 to 1405137 tid 1 2024-09-13T11:32:26.624 INFO:tasks.rados.rados.0.smithi204.stdout:15: writing smithi20438004-15 from 2069689 to 2235298 tid 2 2024-09-13T11:32:26.624 INFO:tasks.rados.rados.0.smithi204.stdout:16: write initial oid 16 2024-09-13T11:32:26.624 INFO:tasks.rados.rados.0.smithi204.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-13T11:32:26.630 INFO:tasks.rados.rados.0.smithi204.stdout:16: writing smithi20438004-16 from 652105 to 1097193 tid 1 2024-09-13T11:32:26.637 INFO:tasks.rados.rados.0.smithi204.stdout:16: writing smithi20438004-16 from 1728171 to 2181159 tid 2 2024-09-13T11:32:26.637 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.637 INFO:tasks.rados.rados.0.smithi204.stdout:1: finishing write tid 1 to smithi20438004-1 2024-09-13T11:32:26.637 INFO:tasks.rados.rados.0.smithi204.stdout:1: finishing write tid 2 to smithi20438004-1 2024-09-13T11:32:26.637 INFO:tasks.rados.rados.0.smithi204.stdout:1: finishing write tid 3 to smithi20438004-1 2024-09-13T11:32:26.637 INFO:tasks.rados.rados.0.smithi204.stdout:1: finishing write tid 4 to smithi20438004-1 2024-09-13T11:32:26.638 INFO:tasks.rados.rados.0.smithi204.stdout:1: oid 1 updating version 0 to 2 2024-09-13T11:32:26.638 INFO:tasks.rados.rados.0.smithi204.stdout:1: oid 1 version 2 is already newer than 1 2024-09-13T11:32:26.638 INFO:tasks.rados.rados.0.smithi204.stdout:1: oid 1 updating version 2 to 3 2024-09-13T11:32:26.638 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:26.638 INFO:tasks.rados.rados.0.smithi204.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-13T11:32:26.638 INFO:tasks.rados.rados.0.smithi204.stdout:2: finishing write tid 1 to smithi20438004-2 2024-09-13T11:32:26.639 INFO:tasks.rados.rados.0.smithi204.stdout:2: finishing write tid 2 to smithi20438004-2 2024-09-13T11:32:26.639 INFO:tasks.rados.rados.0.smithi204.stdout:2: finishing write tid 3 to smithi20438004-2 2024-09-13T11:32:26.639 INFO:tasks.rados.rados.0.smithi204.stdout:1: done (15 left) 2024-09-13T11:32:26.639 INFO:tasks.rados.rados.0.smithi204.stdout:17: write initial oid 17 2024-09-13T11:32:26.639 INFO:tasks.rados.rados.0.smithi204.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-13T11:32:26.643 INFO:tasks.rados.rados.0.smithi204.stdout:17: writing smithi20438004-17 from 668912 to 1305602 tid 1 2024-09-13T11:32:26.644 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:26.645 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:26.645 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:26.645 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:26.649 INFO:tasks.rados.rados.0.smithi204.stdout:17: writing smithi20438004-17 from 1903006 to 2268912 tid 2 2024-09-13T11:32:26.649 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.649 INFO:tasks.rados.rados.0.smithi204.stdout:2: finishing write tid 4 to smithi20438004-2 2024-09-13T11:32:26.649 INFO:tasks.rados.rados.0.smithi204.stdout:2: oid 2 updating version 0 to 2 2024-09-13T11:32:26.649 INFO:tasks.rados.rados.0.smithi204.stdout:2: oid 2 version 2 is already newer than 1 2024-09-13T11:32:26.649 INFO:tasks.rados.rados.0.smithi204.stdout:2: oid 2 updating version 2 to 3 2024-09-13T11:32:26.650 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:26.650 INFO:tasks.rados.rados.0.smithi204.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:26.650 INFO:tasks.rados.rados.0.smithi204.stdout:2: done (15 left) 2024-09-13T11:32:26.650 INFO:tasks.rados.rados.0.smithi204.stdout:18: write initial oid 18 2024-09-13T11:32:26.650 INFO:tasks.rados.rados.0.smithi204.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-13T11:32:26.655 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:32:26.656 INFO:tasks.rados.rados.0.smithi204.stdout:18: writing smithi20438004-18 from 685719 to 1397658 tid 1 2024-09-13T11:32:26.661 INFO:tasks.rados.rados.0.smithi204.stdout:18: writing smithi20438004-18 from 1845135 to 2285719 tid 2 2024-09-13T11:32:26.661 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.661 INFO:tasks.rados.rados.0.smithi204.stdout:3: finishing write tid 1 to smithi20438004-3 2024-09-13T11:32:26.661 INFO:tasks.rados.rados.0.smithi204.stdout:3: finishing write tid 2 to smithi20438004-3 2024-09-13T11:32:26.661 INFO:tasks.rados.rados.0.smithi204.stdout:3: finishing write tid 3 to smithi20438004-3 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:3: finishing write tid 4 to smithi20438004-3 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:3: oid 3 updating version 0 to 2 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:3: oid 3 updating version 2 to 3 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:3: oid 3 version 3 is already newer than 1 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:4: finishing write tid 1 to smithi20438004-4 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:4: finishing write tid 2 to smithi20438004-4 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:3: done (15 left) 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:19: write initial oid 19 2024-09-13T11:32:26.662 INFO:tasks.rados.rados.0.smithi204.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-13T11:32:26.669 INFO:tasks.rados.rados.0.smithi204.stdout:19: writing smithi20438004-19 from 702526 to 1489714 tid 1 2024-09-13T11:32:26.674 INFO:tasks.rados.rados.0.smithi204.stdout:19: writing smithi20438004-19 from 1903617 to 2302526 tid 2 2024-09-13T11:32:26.674 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.674 INFO:tasks.rados.rados.0.smithi204.stdout:4: finishing write tid 3 to smithi20438004-4 2024-09-13T11:32:26.674 INFO:tasks.rados.rados.0.smithi204.stdout:4: finishing write tid 4 to smithi20438004-4 2024-09-13T11:32:26.674 INFO:tasks.rados.rados.0.smithi204.stdout:4: oid 4 updating version 0 to 6 2024-09-13T11:32:26.674 INFO:tasks.rados.rados.0.smithi204.stdout:4: oid 4 version 6 is already newer than 4 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:4: oid 4 version 6 is already newer than 5 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:5: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:5: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:4: done (15 left) 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:20: write initial oid 20 2024-09-13T11:32:26.675 INFO:tasks.rados.rados.0.smithi204.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-13T11:32:26.680 INFO:tasks.rados.rados.0.smithi204.stdout:20: writing smithi20438004-20 from 719333 to 1181770 tid 1 2024-09-13T11:32:26.681 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:26.679+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:20: writing smithi20438004-20 from 1962099 to 2319333 tid 2 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:5: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:5: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:5: oid 5 updating version 0 to 2 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:5: oid 5 updating version 2 to 3 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:5: oid 5 version 3 is already newer than 1 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:6: finishing write tid 1 to smithi20438004-6 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:6: finishing write tid 2 to smithi20438004-6 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:6: finishing write tid 3 to smithi20438004-6 2024-09-13T11:32:26.687 INFO:tasks.rados.rados.0.smithi204.stdout:6: finishing write tid 4 to smithi20438004-6 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:6: oid 6 updating version 0 to 6 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:6: oid 6 version 6 is already newer than 4 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:6: oid 6 version 6 is already newer than 5 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:7: finishing write tid 1 to smithi20438004-7 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:5: done (15 left) 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:6: done (14 left) 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:21: write initial oid 21 2024-09-13T11:32:26.688 INFO:tasks.rados.rados.0.smithi204.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-13T11:32:26.693 INFO:tasks.rados.rados.0.smithi204.stdout:21: writing smithi20438004-21 from 736140 to 1273826 tid 1 2024-09-13T11:32:26.699 INFO:tasks.rados.rados.0.smithi204.stdout:21: writing smithi20438004-21 from 2020581 to 2336140 tid 2 2024-09-13T11:32:26.699 INFO:tasks.rados.rados.0.smithi204.stdout:22: write initial oid 22 2024-09-13T11:32:26.699 INFO:tasks.rados.rados.0.smithi204.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-13T11:32:26.706 INFO:tasks.rados.rados.0.smithi204.stdout:22: writing smithi20438004-22 from 752947 to 1365882 tid 1 2024-09-13T11:32:26.711 INFO:tasks.rados.rados.0.smithi204.stdout:22: writing smithi20438004-22 from 1962710 to 2352947 tid 2 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: finishing write tid 2 to smithi20438004-7 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: finishing write tid 3 to smithi20438004-7 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: finishing write tid 4 to smithi20438004-7 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: oid 7 updating version 0 to 3 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: oid 7 version 3 is already newer than 1 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: oid 7 version 3 is already newer than 2 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:8: finishing write tid 1 to smithi20438004-8 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:8: finishing write tid 2 to smithi20438004-8 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:8: finishing write tid 3 to smithi20438004-8 2024-09-13T11:32:26.712 INFO:tasks.rados.rados.0.smithi204.stdout:7: done (15 left) 2024-09-13T11:32:26.713 INFO:tasks.rados.rados.0.smithi204.stdout:23: write initial oid 23 2024-09-13T11:32:26.713 INFO:tasks.rados.rados.0.smithi204.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-13T11:32:26.719 INFO:tasks.rados.rados.0.smithi204.stdout:23: writing smithi20438004-23 from 769754 to 1457938 tid 1 2024-09-13T11:32:26.724 INFO:tasks.rados.rados.0.smithi204.stdout:23: writing smithi20438004-23 from 2021192 to 2369754 tid 2 2024-09-13T11:32:26.724 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.724 INFO:tasks.rados.rados.0.smithi204.stdout:8: finishing write tid 4 to smithi20438004-8 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:8: oid 8 updating version 0 to 5 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:8: oid 8 updating version 5 to 6 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:8: oid 8 version 6 is already newer than 4 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:9: finishing write tid 1 to smithi20438004-9 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:8: done (15 left) 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:24: write initial oid 24 2024-09-13T11:32:26.725 INFO:tasks.rados.rados.0.smithi204.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-13T11:32:26.731 INFO:tasks.rados.rados.0.smithi204.stdout:24: writing smithi20438004-24 from 786561 to 1266347 tid 1 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:24: writing smithi20438004-24 from 1796027 to 2263691 tid 2 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:9: finishing write tid 2 to smithi20438004-9 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:9: finishing write tid 3 to smithi20438004-9 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:9: finishing write tid 4 to smithi20438004-9 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:9: oid 9 updating version 0 to 3 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:9: oid 9 version 3 is already newer than 2 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:9: oid 9 version 3 is already newer than 1 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:26.737 INFO:tasks.rados.rados.0.smithi204.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: finishing write tid 1 to smithi20438004-10 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: finishing write tid 2 to smithi20438004-10 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: finishing write tid 3 to smithi20438004-10 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: finishing write tid 4 to smithi20438004-10 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: oid 10 updating version 0 to 3 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: oid 10 version 3 is already newer than 2 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: oid 10 version 3 is already newer than 1 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:9: done (15 left) 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:10: done (14 left) 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:25: write initial oid 25 2024-09-13T11:32:26.738 INFO:tasks.rados.rados.0.smithi204.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-13T11:32:26.741 INFO:tasks.rados.rados.0.smithi204.stdout:25: writing smithi20438004-25 from 403368 to 958403 tid 1 2024-09-13T11:32:26.748 INFO:tasks.rados.rados.0.smithi204.stdout:25: writing smithi20438004-25 from 1454509 to 2182184 tid 2 2024-09-13T11:32:26.748 INFO:tasks.rados.rados.0.smithi204.stdout:26: write initial oid 26 2024-09-13T11:32:26.748 INFO:tasks.rados.rados.0.smithi204.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-13T11:32:26.753 INFO:tasks.rados.rados.0.smithi204.stdout:26: writing smithi20438004-26 from 420175 to 1050459 tid 1 2024-09-13T11:32:26.761 INFO:tasks.rados.rados.0.smithi204.stdout:26: writing smithi20438004-26 from 1796638 to 2267971 tid 2 2024-09-13T11:32:26.761 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.761 INFO:tasks.rados.rados.0.smithi204.stdout:11: finishing write tid 1 to smithi20438004-11 2024-09-13T11:32:26.761 INFO:tasks.rados.rados.0.smithi204.stdout:11: finishing write tid 2 to smithi20438004-11 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:11: finishing write tid 3 to smithi20438004-11 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:11: finishing write tid 4 to smithi20438004-11 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:11: oid 11 updating version 0 to 1 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:11: oid 11 updating version 1 to 3 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:11: oid 11 version 3 is already newer than 2 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:12: finishing write tid 1 to smithi20438004-12 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:12: finishing write tid 2 to smithi20438004-12 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:12: finishing write tid 3 to smithi20438004-12 2024-09-13T11:32:26.762 INFO:tasks.rados.rados.0.smithi204.stdout:12: finishing write tid 4 to smithi20438004-12 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:12: oid 12 updating version 0 to 1 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:12: oid 12 updating version 1 to 3 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:12: oid 12 version 3 is already newer than 2 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:13: finishing write tid 1 to smithi20438004-13 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:13: finishing write tid 2 to smithi20438004-13 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:13: finishing write tid 3 to smithi20438004-13 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:11: done (15 left) 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:12: done (14 left) 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:27: write initial oid 27 2024-09-13T11:32:26.763 INFO:tasks.rados.rados.0.smithi204.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-13T11:32:26.767 INFO:tasks.rados.rados.0.smithi204.stdout:27: writing smithi20438004-27 from 436982 to 1142515 tid 1 2024-09-13T11:32:26.767 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:26.767 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:26.767 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:26.767 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:26.767 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:26.774 INFO:tasks.rados.rados.0.smithi204.stdout:27: writing smithi20438004-27 from 1855120 to 2436982 tid 2 2024-09-13T11:32:26.774 INFO:tasks.rados.rados.0.smithi204.stdout:28: write initial oid 28 2024-09-13T11:32:26.774 INFO:tasks.rados.rados.0.smithi204.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-13T11:32:26.777 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:32:26.780 INFO:tasks.rados.rados.0.smithi204.stdout:28: writing smithi20438004-28 from 453789 to 1234571 tid 1 2024-09-13T11:32:26.781 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:26.780+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:28: writing smithi20438004-28 from 1913602 to 2388604 tid 2 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:13: finishing write tid 4 to smithi20438004-13 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:13: oid 13 updating version 0 to 4 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:13: oid 13 updating version 4 to 5 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:13: oid 13 updating version 5 to 6 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:13: done (15 left) 2024-09-13T11:32:26.787 INFO:tasks.rados.rados.0.smithi204.stdout:29: write initial oid 29 2024-09-13T11:32:26.788 INFO:tasks.rados.rados.0.smithi204.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-13T11:32:26.792 INFO:tasks.rados.rados.0.smithi204.stdout:29: writing smithi20438004-29 from 470596 to 926627 tid 1 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout:29: writing smithi20438004-29 from 1572084 to 2190744 tid 2 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout:14: finishing write tid 1 to smithi20438004-14 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout:14: finishing write tid 2 to smithi20438004-14 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout:14: finishing write tid 3 to smithi20438004-14 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout:14: finishing write tid 4 to smithi20438004-14 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout:14: oid 14 updating version 0 to 5 2024-09-13T11:32:26.799 INFO:tasks.rados.rados.0.smithi204.stdout:14: oid 14 version 5 is already newer than 4 2024-09-13T11:32:26.800 INFO:tasks.rados.rados.0.smithi204.stdout:14: oid 14 updating version 5 to 6 2024-09-13T11:32:26.800 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:26.800 INFO:tasks.rados.rados.0.smithi204.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-13T11:32:26.800 INFO:tasks.rados.rados.0.smithi204.stdout:15: finishing write tid 1 to smithi20438004-15 2024-09-13T11:32:26.800 INFO:tasks.rados.rados.0.smithi204.stdout:15: finishing write tid 2 to smithi20438004-15 2024-09-13T11:32:26.800 INFO:tasks.rados.rados.0.smithi204.stdout:15: finishing write tid 3 to smithi20438004-15 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:15: finishing write tid 4 to smithi20438004-15 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:15: oid 15 updating version 0 to 2 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:15: oid 15 updating version 2 to 3 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:15: oid 15 version 3 is already newer than 1 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:14: done (15 left) 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:15: done (14 left) 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:30: write initial oid 30 2024-09-13T11:32:26.801 INFO:tasks.rados.rados.0.smithi204.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-13T11:32:26.804 INFO:tasks.rados.rados.0.smithi204.stdout:30: writing smithi20438004-30 from 487403 to 1018683 tid 1 2024-09-13T11:32:26.809 INFO:tasks.rados.rados.0.smithi204.stdout:30: writing smithi20438004-30 from 1514213 to 1992884 tid 2 2024-09-13T11:32:26.810 INFO:tasks.rados.rados.0.smithi204.stdout:31: write initial oid 31 2024-09-13T11:32:26.810 INFO:tasks.rados.rados.0.smithi204.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-13T11:32:26.815 INFO:tasks.rados.rados.0.smithi204.stdout:31: writing smithi20438004-31 from 504210 to 1110739 tid 1 2024-09-13T11:32:26.823 INFO:tasks.rados.rados.0.smithi204.stdout:31: writing smithi20438004-31 from 1572695 to 2195024 tid 2 2024-09-13T11:32:26.823 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.823 INFO:tasks.rados.rados.0.smithi204.stdout:17: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:26.823 INFO:tasks.rados.rados.0.smithi204.stdout:17: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:26.823 INFO:tasks.rados.rados.0.smithi204.stdout:17: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:17: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:17: oid 17 updating version 0 to 5 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:17: oid 17 version 5 is already newer than 4 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:17: oid 17 updating version 5 to 6 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:18: finishing write tid 1 to smithi20438004-18 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:18: finishing write tid 2 to smithi20438004-18 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:18: finishing write tid 3 to smithi20438004-18 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:18: finishing write tid 4 to smithi20438004-18 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:18: oid 18 updating version 0 to 5 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:18: oid 18 version 5 is already newer than 4 2024-09-13T11:32:26.824 INFO:tasks.rados.rados.0.smithi204.stdout:18: oid 18 updating version 5 to 6 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: finishing write tid 1 to smithi20438004-19 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: finishing write tid 2 to smithi20438004-19 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: finishing write tid 3 to smithi20438004-19 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: finishing write tid 4 to smithi20438004-19 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: oid 19 updating version 0 to 3 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: oid 19 version 3 is already newer than 1 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: oid 19 version 3 is already newer than 2 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-13T11:32:26.825 INFO:tasks.rados.rados.0.smithi204.stdout:20: finishing write tid 1 to smithi20438004-20 2024-09-13T11:32:26.826 INFO:tasks.rados.rados.0.smithi204.stdout:20: finishing write tid 2 to smithi20438004-20 2024-09-13T11:32:26.826 INFO:tasks.rados.rados.0.smithi204.stdout:20: finishing write tid 3 to smithi20438004-20 2024-09-13T11:32:26.826 INFO:tasks.rados.rados.0.smithi204.stdout:17: done (15 left) 2024-09-13T11:32:26.826 INFO:tasks.rados.rados.0.smithi204.stdout:18: done (14 left) 2024-09-13T11:32:26.826 INFO:tasks.rados.rados.0.smithi204.stdout:19: done (13 left) 2024-09-13T11:32:26.826 INFO:tasks.rados.rados.0.smithi204.stdout:32: write initial oid 32 2024-09-13T11:32:26.826 INFO:tasks.rados.rados.0.smithi204.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-13T11:32:26.829 INFO:tasks.rados.rados.0.smithi204.stdout:32: writing smithi20438004-32 from 521017 to 1319148 tid 1 2024-09-13T11:32:26.834 INFO:tasks.rados.rados.0.smithi204.stdout:32: writing smithi20438004-32 from 1747530 to 2229870 tid 2 2024-09-13T11:32:26.835 INFO:tasks.rados.rados.0.smithi204.stdout:33: write initial oid 33 2024-09-13T11:32:26.835 INFO:tasks.rados.rados.0.smithi204.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-13T11:32:26.840 INFO:tasks.rados.rados.0.smithi204.stdout:33: writing smithi20438004-33 from 537824 to 1011204 tid 1 2024-09-13T11:32:26.848 INFO:tasks.rados.rados.0.smithi204.stdout:33: writing smithi20438004-33 from 1806012 to 2432010 tid 2 2024-09-13T11:32:26.848 INFO:tasks.rados.rados.0.smithi204.stdout:34: write initial oid 34 2024-09-13T11:32:26.848 INFO:tasks.rados.rados.0.smithi204.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-13T11:32:26.853 INFO:tasks.rados.rados.0.smithi204.stdout:34: writing smithi20438004-34 from 554631 to 1103260 tid 1 2024-09-13T11:32:26.860 INFO:tasks.rados.rados.0.smithi204.stdout:34: writing smithi20438004-34 from 1748141 to 2234150 tid 2 2024-09-13T11:32:26.860 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.860 INFO:tasks.rados.rados.0.smithi204.stdout:20: finishing write tid 4 to smithi20438004-20 2024-09-13T11:32:26.860 INFO:tasks.rados.rados.0.smithi204.stdout:20: oid 20 updating version 0 to 6 2024-09-13T11:32:26.860 INFO:tasks.rados.rados.0.smithi204.stdout:20: oid 20 version 6 is already newer than 4 2024-09-13T11:32:26.860 INFO:tasks.rados.rados.0.smithi204.stdout:20: oid 20 version 6 is already newer than 5 2024-09-13T11:32:26.861 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:26.861 INFO:tasks.rados.rados.0.smithi204.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:26.861 INFO:tasks.rados.rados.0.smithi204.stdout:21: finishing write tid 1 to smithi20438004-21 2024-09-13T11:32:26.861 INFO:tasks.rados.rados.0.smithi204.stdout:21: finishing write tid 2 to smithi20438004-21 2024-09-13T11:32:26.861 INFO:tasks.rados.rados.0.smithi204.stdout:20: done (15 left) 2024-09-13T11:32:26.861 INFO:tasks.rados.rados.0.smithi204.stdout:35: write initial oid 35 2024-09-13T11:32:26.861 INFO:tasks.rados.rados.0.smithi204.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-13T11:32:26.866 INFO:tasks.rados.rados.0.smithi204.stdout:35: writing smithi20438004-35 from 571438 to 1195316 tid 1 2024-09-13T11:32:26.873 INFO:tasks.rados.rados.0.smithi204.stdout:35: writing smithi20438004-35 from 1806623 to 2436290 tid 2 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:21: finishing write tid 3 to smithi20438004-21 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:21: finishing write tid 4 to smithi20438004-21 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:21: oid 21 updating version 0 to 1 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:21: oid 21 updating version 1 to 2 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:21: oid 21 updating version 2 to 3 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:22: finishing write tid 1 to smithi20438004-22 2024-09-13T11:32:26.874 INFO:tasks.rados.rados.0.smithi204.stdout:22: finishing write tid 2 to smithi20438004-22 2024-09-13T11:32:26.875 INFO:tasks.rados.rados.0.smithi204.stdout:22: finishing write tid 3 to smithi20438004-22 2024-09-13T11:32:26.875 INFO:tasks.rados.rados.0.smithi204.stdout:21: done (15 left) 2024-09-13T11:32:26.875 INFO:tasks.rados.rados.0.smithi204.stdout:36: write initial oid 36 2024-09-13T11:32:26.875 INFO:tasks.rados.rados.0.smithi204.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-13T11:32:26.880 INFO:tasks.rados.rados.0.smithi204.stdout:36: writing smithi20438004-36 from 588245 to 1287372 tid 1 2024-09-13T11:32:26.882 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:26.881+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:26.886 INFO:tasks.rados.rados.0.smithi204.stdout:36: writing smithi20438004-36 from 1865105 to 2354783 tid 2 2024-09-13T11:32:26.886 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:22: finishing write tid 4 to smithi20438004-22 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:22: oid 22 updating version 0 to 7 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:22: oid 22 updating version 7 to 9 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:22: oid 22 version 9 is already newer than 8 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:22: done (15 left) 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:37: write initial oid 37 2024-09-13T11:32:26.887 INFO:tasks.rados.rados.0.smithi204.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-13T11:32:26.889 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.515008388999998, 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27446322899999998, 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.890 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1100000000000005e-07 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372244+0000", 2024-09-13T11:32:26.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.66e-06 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372256+0000", 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1817000000000001e-05 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372264+0000", 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0220000000000004e-06 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372320+0000", 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6437999999999997e-05 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372329+0000", 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644121+0000", 2024-09-13T11:32:26.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27179224899999999 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644271+0000", 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014990000000000001 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644421+0000", 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015035600000000001 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646446+0000", 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00202454 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646524+0000", 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8839999999999997e-05 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646598+0000", 2024-09-13T11:32:26.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3862999999999994e-05 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646654+0000", 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5609999999999998e-05 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646701+0000", 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6844000000000001e-05 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.897 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.777318+0000", 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.109927457000001, 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010370683, 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 48 2024-09-13T11:32:26.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777318+0000", 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777318+0000", 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777319+0000", 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0100000000000005e-07 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777321+0000", 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.531e-06 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777325+0000", 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2310000000000001e-06 2024-09-13T11:32:26.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777328+0000", 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.909e-06 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.782359+0000", 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050316029999999999 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.782438+0000", 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9089000000000001e-05 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.782566+0000", 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012767900000000001 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.785100+0000", 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025336439999999998 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.786813+0000", 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017137949999999999 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787594+0000", 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078046500000000004 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787650+0000", 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6029000000000002e-05 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787689+0000", 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9206999999999999e-05 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.777331+0000", 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.109914510999999, 2024-09-13T11:32:26.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010437329, 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 49 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777331+0000", 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777331+0000", 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777332+0000", 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4699999999999998e-07 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777334+0000", 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.013e-06 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777338+0000", 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8199999999999996e-06 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777342+0000", 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.636e-06 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.782947+0000", 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005604777 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.783000+0000", 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2954999999999997e-05 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.783084+0000", 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4330000000000001e-05 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.785106+0000", 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020218050000000002 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.786851+0000", 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017447809999999999 2024-09-13T11:32:26.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787717+0000", 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00086670599999999997 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787746+0000", 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8758000000000001e-05 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787769+0000", 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2501e-05 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.777355+0000", 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.109890557, 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010552311999999999, 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.911 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 50 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777355+0000", 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777355+0000", 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777356+0000", 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6599999999999996e-07 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777407+0000", 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0976000000000001e-05 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777416+0000", 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8859999999999993e-06 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777423+0000", 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7109999999999995e-06 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.783433+0000", 2024-09-13T11:32:26.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060099680000000001 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.783509+0000", 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5673000000000005e-05 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.783637+0000", 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012807199999999999 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.785152+0000", 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015143450000000001 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.786976+0000", 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018246340000000001 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787818+0000", 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000842091 2024-09-13T11:32:26.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787870+0000", 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1508999999999998e-05 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787908+0000", 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7880999999999999e-05 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.777433+0000", 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.109812688000002, 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010566320000000001, 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.918 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 51 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777433+0000", 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777433+0000", 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777434+0000", 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.46e-07 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777436+0000", 2024-09-13T11:32:26.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6690000000000002e-06 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777441+0000", 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4599999999999996e-06 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777444+0000", 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.619e-06 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.784104+0000", 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0066596169999999996 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.784158+0000", 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.784295+0000", 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013655899999999999 2024-09-13T11:32:26.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.785157+0000", 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00086217100000000001 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787017+0000", 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018596439999999999 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787954+0000", 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093734999999999999 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.787981+0000", 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6676999999999999e-05 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.788000+0000", 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.897e-05 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.777449+0000", 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.109797403999998, 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012476724, 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.925 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 52 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777449+0000", 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777449+0000", 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777449+0000", 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0900000000000002e-07 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777452+0000", 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9069999999999999e-06 2024-09-13T11:32:26.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777455+0000", 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1030000000000002e-06 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.777458+0000", 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3569999999999998e-06 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.784632+0000", 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071733049999999996 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.784709+0000", 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7873999999999993e-05 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.784845+0000", 2024-09-13T11:32:26.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013520700000000001 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.785288+0000", 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044357000000000001 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.788047+0000", 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027588040000000001 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.789834+0000", 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017867600000000001 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.789888+0000", 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3872000000000002e-05 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.789925+0000", 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7555999999999998e-05 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.981289+0000", 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.905956490000001, 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010300461, 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 97 2024-09-13T11:32:26.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981289+0000", 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981289+0000", 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981290+0000", 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2800000000000003e-07 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981292+0000", 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0329999999999998e-06 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981296+0000", 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6310000000000002e-06 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981299+0000", 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5749999999999999e-06 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.985202+0000", 2024-09-13T11:32:26.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039034510000000001 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.985253+0000", 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0648e-05 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.985323+0000", 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0055999999999995e-05 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.987170+0000", 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00184651 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.988521+0000", 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001351435 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991547+0000", 2024-09-13T11:32:26.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030257370000000001 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991571+0000", 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4229000000000001e-05 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991590+0000", 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8828e-05 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.940 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.981304+0000", 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.905942089, 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010382206999999999, 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.941 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.944 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.944 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.944 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 98 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981304+0000", 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981304+0000", 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981304+0000", 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.16e-07 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981306+0000", 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.844e-06 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981309+0000", 2024-09-13T11:32:26.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.055e-06 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981311+0000", 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5500000000000001e-06 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.985714+0000", 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044021620000000003 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.985771+0000", 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7664999999999998e-05 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.985843+0000", 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2107000000000006e-05 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.987174+0000", 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013310559999999999 2024-09-13T11:32:26.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.988546+0000", 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001371693 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991625+0000", 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030789519999999998 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991652+0000", 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6747999999999999e-05 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991686+0000", 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4159000000000003e-05 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.950 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.981315+0000", 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.905930861000002, 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010435926, 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 99 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981315+0000", 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981315+0000", 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981315+0000", 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.16e-07 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981346+0000", 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1009000000000001e-05 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981350+0000", 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8539999999999999e-06 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981354+0000", 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2300000000000002e-06 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.986188+0000", 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048337570000000002 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.986270+0000", 2024-09-13T11:32:26.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2171999999999999e-05 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.986398+0000", 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012726600000000001 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.987183+0000", 2024-09-13T11:32:26.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078498299999999995 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.988644+0000", 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001461838 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991709+0000", 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003064995 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991733+0000", 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4026999999999999e-05 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991751+0000", 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7578999999999999e-05 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.981359+0000", 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.905886696, 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010503764000000001, 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 100 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981359+0000", 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981359+0000", 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981360+0000", 2024-09-13T11:32:26.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2300000000000002e-07 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981363+0000", 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3969999999999998e-06 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981367+0000", 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9140000000000001e-06 2024-09-13T11:32:26.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.981370+0000", 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4010000000000001e-06 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.986733+0000", 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053625260000000003 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.986777+0000", 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4224999999999998e-05 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.986847+0000", 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0131999999999999e-05 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.987395+0000", 2024-09-13T11:32:26.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054773599999999995 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.988932+0000", 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015373120000000001 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991804+0000", 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028718699999999999 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991830+0000", 2024-09-13T11:32:26.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5717000000000001e-05 2024-09-13T11:32:26.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.991863+0000", 2024-09-13T11:32:26.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3210999999999998e-05 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.068808+0000", 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818437418999999, 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012722091, 2024-09-13T11:32:26.965 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 115 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068808+0000", 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068808+0000", 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068809+0000", 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9200000000000002e-07 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068811+0000", 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2089999999999999e-06 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068815+0000", 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0899999999999998e-06 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068818+0000", 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7489999999999999e-06 2024-09-13T11:32:26.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.070305+0000", 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014874739999999999 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.070348+0000", 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3058e-05 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.070423+0000", 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4128000000000006e-05 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.071850+0000", 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00142731 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073766+0000", 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019164469999999999 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.081477+0000", 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007710901 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.081508+0000", 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0417e-05 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.081531+0000", 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2915999999999999e-05 2024-09-13T11:32:26.971 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.068899+0000", 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818347261, 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012714685, 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.972 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 116 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068899+0000", 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068899+0000", 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068899+0000", 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5400000000000001e-07 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068902+0000", 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9900000000000002e-06 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068915+0000", 2024-09-13T11:32:26.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2588e-05 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068919+0000", 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0679999999999998e-06 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.070821+0000", 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019021330000000001 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.070894+0000", 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3337000000000006e-05 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.070978+0000", 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4085999999999994e-05 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.072694+0000", 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00171569 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073797+0000", 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001103312 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.081563+0000", 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077651600000000001 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.081592+0000", 2024-09-13T11:32:26.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9017999999999999e-05 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.081613+0000", 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1749000000000001e-05 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.068950+0000", 2024-09-13T11:32:26.978 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818295939999999, 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015718361, 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 117 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068950+0000", 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068950+0000", 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.068957+0000", 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5009999999999998e-06 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069033+0000", 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5140000000000002e-05 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069039+0000", 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1850000000000001e-06 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069049+0000", 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0295000000000001e-05 2024-09-13T11:32:26.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.071320+0000", 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002270977 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.071367+0000", 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7209000000000002e-05 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.071446+0000", 2024-09-13T11:32:26.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8343999999999994e-05 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073593+0000", 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021471400000000001 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074727+0000", 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011344459999999999 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.084562+0000", 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098344220000000006 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.084645+0000", 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3098999999999998e-05 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.084668+0000", 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3603000000000001e-05 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.984 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:118 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.069057+0000", 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818188622000001, 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015942426999999999, 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.985 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 118 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069057+0000", 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069057+0000", 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069058+0000", 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.96e-07 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069061+0000", 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9560000000000002e-06 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069066+0000", 2024-09-13T11:32:26.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5539999999999997e-06 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069078+0000", 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2116000000000001e-05 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.071875+0000", 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027962260000000002 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.071920+0000", 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5334999999999999e-05 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.071989+0000", 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8853e-05 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073596+0000", 2024-09-13T11:32:26.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016075379999999999 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074758+0000", 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001161446 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.084952+0000", 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010193817000000001 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.084980+0000", 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8158999999999999e-05 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085000+0000", 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0030999999999998e-05 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.991 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:119 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.069084+0000", 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818162394000002, 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016001393999999999, 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 119 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069084+0000", 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069084+0000", 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069084+0000", 2024-09-13T11:32:26.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7099999999999998e-07 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069087+0000", 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0079999999999998e-06 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069090+0000", 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2480000000000001e-06 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069093+0000", 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8449999999999999e-06 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.072310+0000", 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032172910000000002 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.072354+0000", 2024-09-13T11:32:26.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3621000000000001e-05 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.072426+0000", 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1730000000000006e-05 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073735+0000", 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001309223 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074788+0000", 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010533809999999999 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085038+0000", 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010250356 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085064+0000", 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5989000000000001e-05 2024-09-13T11:32:26.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085085+0000", 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0431000000000001e-05 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:120 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.069109+0000", 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818136473999999, 2024-09-13T11:32:26.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016050512999999999, 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 120 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069109+0000", 2024-09-13T11:32:26.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069109+0000", 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069110+0000", 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7300000000000002e-07 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069171+0000", 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1575000000000006e-05 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069179+0000", 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-06 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069185+0000", 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9299999999999997e-06 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.072721+0000", 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035353329999999999 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.072766+0000", 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5673999999999997e-05 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.072839+0000", 2024-09-13T11:32:27.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2154000000000001e-05 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073738+0000", 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089973400000000002 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074817+0000", 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010783959999999999 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085115+0000", 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010298567999999999 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085140+0000", 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4547999999999999e-05 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085160+0000", 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0058e-05 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.004 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:121 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.069212+0000", 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818033445000001, 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016018983000000001, 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:27.005 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 121 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069212+0000", 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069212+0000", 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069213+0000", 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1900000000000002e-07 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069218+0000", 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7929999999999997e-06 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069227+0000", 2024-09-13T11:32:27.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8869999999999998e-06 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069232+0000", 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3730000000000001e-06 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073212+0000", 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039794920000000003 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073256+0000", 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4273000000000001e-05 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073322+0000", 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6109e-05 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074765+0000", 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001442718 2024-09-13T11:32:27.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074849+0000", 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4324000000000001e-05 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085187+0000", 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010338053999999999 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085211+0000", 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3952e-05 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.085231+0000", 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0389000000000001e-05 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:27.011 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.069244+0000", 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.818002379999999, 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018133092, 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 122 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069244+0000", 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069244+0000", 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069244+0000", 2024-09-13T11:32:27.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5700000000000002e-07 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069247+0000", 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3340000000000002e-06 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069253+0000", 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5709999999999997e-06 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069258+0000", 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.843e-06 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073830+0000", 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045719109999999997 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073878+0000", 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7938000000000001e-05 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.073955+0000", 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7773000000000002e-05 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074768+0000", 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081253100000000002 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.075485+0000", 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00071657600000000002 2024-09-13T11:32:27.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087323+0000", 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011838402 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087353+0000", 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9736000000000001e-05 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:27.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087377+0000", 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3920000000000001e-05 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.069273+0000", 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.817973213999998, 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018329593000000002, 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 123 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069273+0000", 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069273+0000", 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069273+0000", 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069320+0000", 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7253999999999999e-05 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069328+0000", 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7430000000000002e-06 2024-09-13T11:32:27.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069334+0000", 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8690000000000001e-06 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074294+0000", 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049595239999999999 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074339+0000", 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5262999999999997e-05 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074408+0000", 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9070000000000001e-05 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.075236+0000", 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082819999999999996 2024-09-13T11:32:27.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.075621+0000", 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038493000000000002 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087550+0000", 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011929422 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087581+0000", 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0530000000000001e-05 2024-09-13T11:32:27.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087602+0000", 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1288999999999999e-05 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:05.069344+0000", 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.817901894999999, 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018336524999999999, 2024-09-13T11:32:27.024 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069344+0000", 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069344+0000", 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069345+0000", 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7000000000000005e-07 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069348+0000", 2024-09-13T11:32:27.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1839999999999999e-06 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069360+0000", 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2489e-05 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.069367+0000", 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3690000000000003e-06 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:27.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074871+0000", 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055039449999999997 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074912+0000", 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1841999999999997e-05 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.074986+0000", 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3163000000000004e-05 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.075400+0000", 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041414700000000002 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.076270+0000", 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000870695 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087633+0000", 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011362956 2024-09-13T11:32:27.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087660+0000", 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:05.087681+0000", 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0959999999999999e-05 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:27.030 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:27.031 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:27.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:32:27.038 INFO:tasks.rados.rados.0.smithi204.stdout:37: writing smithi20438004-37 from 605052 to 1379428 tid 1 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:37: writing smithi20438004-37 from 1923587 to 2556923 tid 2 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: oid 23 updating version 0 to 7 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: oid 23 updating version 7 to 9 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: oid 23 version 9 is already newer than 8 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:24: finishing write tid 1 to smithi20438004-24 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:24: finishing write tid 2 to smithi20438004-24 2024-09-13T11:32:27.039 INFO:tasks.rados.rados.0.smithi204.stdout:24: finishing write tid 3 to smithi20438004-24 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:24: finishing write tid 4 to smithi20438004-24 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:24: oid 24 updating version 0 to 8 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:24: oid 24 version 8 is already newer than 7 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:24: oid 24 updating version 8 to 9 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:25: finishing write tid 1 to smithi20438004-25 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:25: finishing write tid 2 to smithi20438004-25 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:25: finishing write tid 3 to smithi20438004-25 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:25: finishing write tid 4 to smithi20438004-25 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:25: oid 25 updating version 0 to 4 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:25: oid 25 updating version 4 to 6 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:25: oid 25 version 6 is already newer than 5 2024-09-13T11:32:27.040 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: finishing write tid 1 to smithi20438004-26 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: finishing write tid 2 to smithi20438004-26 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: finishing write tid 3 to smithi20438004-26 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: finishing write tid 4 to smithi20438004-26 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: oid 26 updating version 0 to 1 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: oid 26 updating version 1 to 3 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: oid 26 version 3 is already newer than 2 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:27: finishing write tid 1 to smithi20438004-27 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:27: finishing write tid 2 to smithi20438004-27 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:27: finishing write tid 3 to smithi20438004-27 2024-09-13T11:32:27.041 INFO:tasks.rados.rados.0.smithi204.stdout:27: finishing write tid 4 to smithi20438004-27 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:27: oid 27 updating version 0 to 7 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:27: oid 27 updating version 7 to 8 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:27: oid 27 updating version 8 to 9 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: finishing write tid 1 to smithi20438004-28 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: finishing write tid 2 to smithi20438004-28 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: finishing write tid 3 to smithi20438004-28 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: finishing write tid 4 to smithi20438004-28 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: oid 28 updating version 0 to 6 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: oid 28 version 6 is already newer than 4 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: oid 28 version 6 is already newer than 5 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.042 INFO:tasks.rados.rados.0.smithi204.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:29: finishing write tid 1 to smithi20438004-29 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:23: done (15 left) 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:24: done (14 left) 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:25: done (13 left) 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:26: done (12 left) 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:27: done (11 left) 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:28: done (10 left) 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:38: write initial oid 38 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:38: writing smithi20438004-38 from 621859 to 1071484 tid 1 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:38: writing smithi20438004-38 from 1865716 to 2621859 tid 2 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:39: write initial oid 39 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-13T11:32:27.043 INFO:tasks.rados.rados.0.smithi204.stdout:39: writing smithi20438004-39 from 638666 to 1163540 tid 1 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:39: writing smithi20438004-39 from 1924198 to 2561203 tid 2 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:40: write initial oid 40 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:40: writing smithi20438004-40 from 655473 to 1371949 tid 1 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:40: writing smithi20438004-40 from 2099033 to 2655473 tid 2 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:41: write initial oid 41 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:41: writing smithi20438004-41 from 672280 to 1464005 tid 1 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:41: writing smithi20438004-41 from 2157515 to 2672280 tid 2 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:42: write initial oid 42 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:42: writing smithi20438004-42 from 689087 to 1156061 tid 1 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:42: writing smithi20438004-42 from 1699644 to 2483976 tid 2 2024-09-13T11:32:27.044 INFO:tasks.rados.rados.0.smithi204.stdout:43: write initial oid 43 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:43: writing smithi20438004-43 from 705894 to 1248117 tid 1 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:43: writing smithi20438004-43 from 1758126 to 2402469 tid 2 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:29: finishing write tid 2 to smithi20438004-29 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:29: finishing write tid 3 to smithi20438004-29 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:29: finishing write tid 4 to smithi20438004-29 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:29: oid 29 updating version 0 to 12 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:29: oid 29 version 12 is already newer than 11 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:29: oid 29 version 12 is already newer than 10 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-13T11:32:27.045 INFO:tasks.rados.rados.0.smithi204.stdout:30: finishing write tid 1 to smithi20438004-30 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:29: done (15 left) 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:44: write initial oid 44 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:44: writing smithi20438004-44 from 722701 to 1340173 tid 1 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:44: writing smithi20438004-44 from 1816608 to 2604609 tid 2 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:30: finishing write tid 2 to smithi20438004-30 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:30: finishing write tid 3 to smithi20438004-30 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:30: finishing write tid 4 to smithi20438004-30 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:30: oid 30 updating version 0 to 12 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:30: oid 30 version 12 is already newer than 10 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:30: oid 30 version 12 is already newer than 11 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.046 INFO:tasks.rados.rados.0.smithi204.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: finishing write tid 1 to smithi20438004-31 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: finishing write tid 2 to smithi20438004-31 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: finishing write tid 3 to smithi20438004-31 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: finishing write tid 4 to smithi20438004-31 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: oid 31 updating version 0 to 6 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: oid 31 version 6 is already newer than 5 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: oid 31 version 6 is already newer than 4 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:16: finishing write tid 1 to smithi20438004-16 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:16: finishing write tid 2 to smithi20438004-16 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:16: finishing write tid 3 to smithi20438004-16 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:16: finishing write tid 4 to smithi20438004-16 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:16: oid 16 updating version 0 to 3 2024-09-13T11:32:27.047 INFO:tasks.rados.rados.0.smithi204.stdout:16: oid 16 version 3 is already newer than 1 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:16: oid 16 version 3 is already newer than 2 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: finishing write tid 1 to smithi20438004-32 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: finishing write tid 2 to smithi20438004-32 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: finishing write tid 3 to smithi20438004-32 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: finishing write tid 4 to smithi20438004-32 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: oid 32 updating version 0 to 6 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: oid 32 version 6 is already newer than 4 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: oid 32 version 6 is already newer than 5 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:33: finishing write tid 1 to smithi20438004-33 2024-09-13T11:32:27.048 INFO:tasks.rados.rados.0.smithi204.stdout:33: finishing write tid 2 to smithi20438004-33 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:33: finishing write tid 3 to smithi20438004-33 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:33: finishing write tid 4 to smithi20438004-33 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:33: oid 33 updating version 0 to 8 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:33: oid 33 version 8 is already newer than 7 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:33: oid 33 updating version 8 to 9 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:34: finishing write tid 1 to smithi20438004-34 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:34: finishing write tid 2 to smithi20438004-34 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:34: finishing write tid 3 to smithi20438004-34 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:16: done (15 left) 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:30: done (14 left) 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:31: done (13 left) 2024-09-13T11:32:27.049 INFO:tasks.rados.rados.0.smithi204.stdout:32: done (12 left) 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:33: done (11 left) 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:45: write initial oid 45 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:45: writing smithi20438004-45 from 739508 to 1432229 tid 1 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:45: writing smithi20438004-45 from 1875090 to 2523102 tid 2 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:46: write initial oid 46 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:46: writing smithi20438004-46 from 756315 to 1524285 tid 1 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:46: writing smithi20438004-46 from 1933572 to 2725242 tid 2 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:47: write initial oid 47 2024-09-13T11:32:27.050 INFO:tasks.rados.rados.0.smithi204.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-13T11:32:27.051 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:27.030+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:27.051 INFO:tasks.rados.rados.0.smithi204.stdout:47: writing smithi20438004-47 from 773122 to 1332694 tid 1 2024-09-13T11:32:27.051 INFO:tasks.rados.rados.0.smithi204.stdout:47: writing smithi20438004-47 from 1992054 to 2643735 tid 2 2024-09-13T11:32:27.051 INFO:tasks.rados.rados.0.smithi204.stdout:48: write initial oid 48 2024-09-13T11:32:27.051 INFO:tasks.rados.rados.0.smithi204.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-13T11:32:27.051 INFO:tasks.rados.rados.0.smithi204.stdout:48: writing smithi20438004-48 from 789929 to 1424750 tid 1 2024-09-13T11:32:27.057 INFO:tasks.rados.rados.0.smithi204.stdout:48: writing smithi20438004-48 from 2050536 to 2789929 tid 2 2024-09-13T11:32:27.057 INFO:tasks.rados.rados.0.smithi204.stdout:49: write initial oid 49 2024-09-13T11:32:27.057 INFO:tasks.rados.rados.0.smithi204.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-13T11:32:27.063 INFO:tasks.rados.rados.0.smithi204.stdout:49: writing smithi20438004-49 from 406736 to 1116806 tid 1 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout:49: writing smithi20438004-49 from 1709018 to 2364368 tid 2 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout:34: finishing write tid 4 to smithi20438004-34 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout:34: oid 34 updating version 0 to 8 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout:34: oid 34 updating version 8 to 9 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout:34: oid 34 version 9 is already newer than 7 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.070 INFO:tasks.rados.rados.0.smithi204.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-13T11:32:27.071 INFO:tasks.rados.rados.0.smithi204.stdout:35: finishing write tid 1 to smithi20438004-35 2024-09-13T11:32:27.071 INFO:tasks.rados.rados.0.smithi204.stdout:35: finishing write tid 2 to smithi20438004-35 2024-09-13T11:32:27.071 INFO:tasks.rados.rados.0.smithi204.stdout:35: finishing write tid 3 to smithi20438004-35 2024-09-13T11:32:27.071 INFO:tasks.rados.rados.0.smithi204.stdout:34: done (15 left) 2024-09-13T11:32:27.071 INFO:tasks.rados.rados.0.smithi204.stdout:50: write initial oid 50 2024-09-13T11:32:27.071 INFO:tasks.rados.rados.0.smithi204.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-13T11:32:27.076 INFO:tasks.rados.rados.0.smithi204.stdout:50: writing smithi20438004-50 from 423543 to 1208862 tid 1 2024-09-13T11:32:27.091 INFO:tasks.rados.rados.0.smithi204.stdout:50: writing smithi20438004-50 from 1767500 to 2566508 tid 2 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:35: finishing write tid 4 to smithi20438004-35 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:35: oid 35 updating version 0 to 12 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:35: oid 35 version 12 is already newer than 10 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:35: oid 35 version 12 is already newer than 11 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:36: finishing write tid 1 to smithi20438004-36 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:36: finishing write tid 2 to smithi20438004-36 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:36: finishing write tid 3 to smithi20438004-36 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:36: finishing write tid 4 to smithi20438004-36 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:36: oid 36 updating version 0 to 9 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:36: oid 36 version 9 is already newer than 8 2024-09-13T11:32:27.092 INFO:tasks.rados.rados.0.smithi204.stdout:36: oid 36 version 9 is already newer than 7 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:37: finishing write tid 1 to smithi20438004-37 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:35: done (15 left) 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:36: done (14 left) 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:51: copy oid 2 target oid 2 low_tier 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 2 to 2 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:52: copy oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 3 to 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:37: finishing write tid 2 to smithi20438004-37 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:37: finishing write tid 3 to smithi20438004-37 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:37: finishing write tid 4 to smithi20438004-37 2024-09-13T11:32:27.093 INFO:tasks.rados.rados.0.smithi204.stdout:37: oid 37 updating version 0 to 2 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:37: oid 37 updating version 2 to 3 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:37: oid 37 version 3 is already newer than 1 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 3 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: finishing write tid 1 to smithi20438004-38 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: finishing write tid 2 to smithi20438004-38 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: finishing write tid 3 to smithi20438004-38 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: finishing write tid 4 to smithi20438004-38 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: oid 38 updating version 0 to 15 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: oid 38 version 15 is already newer than 14 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: oid 38 version 15 is already newer than 13 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-13T11:32:27.094 INFO:tasks.rados.rados.0.smithi204.stdout:39: finishing write tid 1 to smithi20438004-39 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:37: done (15 left) 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:38: done (14 left) 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:53: copy oid 4 target oid 4 low_tier 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 4 to 4 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:54: copy oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 5 to 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:39: finishing write tid 2 to smithi20438004-39 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:39: finishing write tid 3 to smithi20438004-39 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:39: finishing write tid 4 to smithi20438004-39 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:39: oid 39 updating version 0 to 11 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:39: oid 39 version 11 is already newer than 10 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:39: oid 39 updating version 11 to 12 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.095 INFO:tasks.rados.rados.0.smithi204.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: finishing write tid 1 to smithi20438004-40 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:39: done (15 left) 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:55: copy oid 6 target oid 6 low_tier 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 6 to 6 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: finishing write tid 2 to smithi20438004-40 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: finishing write tid 3 to smithi20438004-40 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: finishing write tid 4 to smithi20438004-40 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: oid 40 updating version 0 to 12 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: oid 40 version 12 is already newer than 10 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: oid 40 version 12 is already newer than 11 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:41: finishing write tid 1 to smithi20438004-41 2024-09-13T11:32:27.096 INFO:tasks.rados.rados.0.smithi204.stdout:41: finishing write tid 2 to smithi20438004-41 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:41: finishing write tid 3 to smithi20438004-41 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:41: finishing write tid 4 to smithi20438004-41 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:41: oid 41 updating version 0 to 15 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:41: oid 41 version 15 is already newer than 13 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:41: oid 41 version 15 is already newer than 14 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:42: finishing write tid 1 to smithi20438004-42 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:42: finishing write tid 2 to smithi20438004-42 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:42: finishing write tid 3 to smithi20438004-42 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:42: finishing write tid 4 to smithi20438004-42 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:42: oid 42 updating version 0 to 11 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:42: oid 42 updating version 11 to 12 2024-09-13T11:32:27.097 INFO:tasks.rados.rados.0.smithi204.stdout:42: oid 42 version 12 is already newer than 10 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:43: finishing write tid 1 to smithi20438004-43 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:43: finishing write tid 2 to smithi20438004-43 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:40: done (15 left) 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:41: done (14 left) 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:42: done (13 left) 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:56: copy oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 7 to 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:57: copy oid 8 target oid 8 low_tier 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 8 to 8 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:58: copy oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 9 to 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.098 INFO:tasks.rados.rados.0.smithi204.stdout:43: finishing write tid 3 to smithi20438004-43 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:43: finishing write tid 4 to smithi20438004-43 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:43: oid 43 updating version 0 to 16 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:43: oid 43 updating version 16 to 18 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:43: oid 43 version 18 is already newer than 17 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 18 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:44: finishing write tid 1 to smithi20438004-44 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:44: finishing write tid 2 to smithi20438004-44 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:44: finishing write tid 3 to smithi20438004-44 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:44: finishing write tid 4 to smithi20438004-44 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:44: oid 44 updating version 0 to 4 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:44: oid 44 updating version 4 to 5 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:44: oid 44 updating version 5 to 6 2024-09-13T11:32:27.099 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:43: done (15 left) 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:44: done (14 left) 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:59: copy oid 10 target oid 10 low_tier 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 10 to 10 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:60: copy oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 11 to 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:45: finishing write tid 1 to smithi20438004-45 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:45: finishing write tid 2 to smithi20438004-45 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:45: finishing write tid 3 to smithi20438004-45 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:45: finishing write tid 4 to smithi20438004-45 2024-09-13T11:32:27.100 INFO:tasks.rados.rados.0.smithi204.stdout:45: oid 45 updating version 0 to 7 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:45: oid 45 updating version 7 to 9 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:45: oid 45 version 9 is already newer than 8 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:46: finishing write tid 1 to smithi20438004-46 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:45: done (15 left) 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:61: copy oid 12 target oid 12 low_tier 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 12 to 12 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:46: finishing write tid 2 to smithi20438004-46 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:46: finishing write tid 3 to smithi20438004-46 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:46: finishing write tid 4 to smithi20438004-46 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:46: oid 46 updating version 0 to 8 2024-09-13T11:32:27.101 INFO:tasks.rados.rados.0.smithi204.stdout:46: oid 46 updating version 8 to 9 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:46: oid 46 version 9 is already newer than 7 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:47: finishing write tid 1 to smithi20438004-47 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:46: done (15 left) 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:62: copy oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 13 to 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:47: finishing write tid 2 to smithi20438004-47 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:47: finishing write tid 3 to smithi20438004-47 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:47: finishing write tid 4 to smithi20438004-47 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:47: oid 47 updating version 0 to 6 2024-09-13T11:32:27.102 INFO:tasks.rados.rados.0.smithi204.stdout:47: oid 47 version 6 is already newer than 4 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:47: oid 47 version 6 is already newer than 5 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:48: finishing write tid 1 to smithi20438004-48 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:47: done (15 left) 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:63: copy oid 14 target oid 14 low_tier 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 14 to 14 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:48: finishing write tid 2 to smithi20438004-48 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:48: finishing write tid 3 to smithi20438004-48 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:48: finishing write tid 4 to smithi20438004-48 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:48: oid 48 updating version 0 to 11 2024-09-13T11:32:27.103 INFO:tasks.rados.rados.0.smithi204.stdout:48: oid 48 version 11 is already newer than 10 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:48: oid 48 updating version 11 to 12 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:49: finishing write tid 1 to smithi20438004-49 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:48: done (15 left) 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:64: copy oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 15 to 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:49: finishing write tid 2 to smithi20438004-49 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:49: finishing write tid 3 to smithi20438004-49 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:49: finishing write tid 4 to smithi20438004-49 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:49: oid 49 updating version 0 to 5 2024-09-13T11:32:27.104 INFO:tasks.rados.rados.0.smithi204.stdout:49: oid 49 updating version 5 to 6 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:49: oid 49 version 6 is already newer than 4 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 6 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:50: finishing write tid 1 to smithi20438004-50 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:49: done (15 left) 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:65: copy oid 16 target oid 16 low_tier 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 16 to 16 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:50: finishing write tid 2 to smithi20438004-50 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:50: finishing write tid 3 to smithi20438004-50 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:50: finishing write tid 4 to smithi20438004-50 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:50: oid 50 updating version 0 to 7 2024-09-13T11:32:27.105 INFO:tasks.rados.rados.0.smithi204.stdout:50: oid 50 updating version 7 to 8 2024-09-13T11:32:27.106 INFO:tasks.rados.rados.0.smithi204.stdout:50: oid 50 updating version 8 to 9 2024-09-13T11:32:27.106 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.106 INFO:tasks.rados.rados.0.smithi204.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-13T11:32:27.106 INFO:tasks.rados.rados.0.smithi204.stdout:50: done (15 left) 2024-09-13T11:32:27.106 INFO:tasks.rados.rados.0.smithi204.stdout:66: copy oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.106 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 17 to 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.106 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.108 INFO:tasks.rados.rados.0.smithi204.stdout:56: finishing copy op to oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.109 INFO:tasks.rados.rados.0.smithi204.stdout:56: done (15 left) 2024-09-13T11:32:27.109 INFO:tasks.rados.rados.0.smithi204.stdout:67: copy oid 18 target oid 18 low_tier 2024-09-13T11:32:27.109 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 18 to 18 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.109 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.110 INFO:tasks.rados.rados.0.smithi204.stdout:54: finishing copy op to oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.111 INFO:tasks.rados.rados.0.smithi204.stdout:54: done (15 left) 2024-09-13T11:32:27.111 INFO:tasks.rados.rados.0.smithi204.stdout:68: copy oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.111 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 19 to 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.111 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.111 INFO:tasks.rados.rados.0.smithi204.stdout:59: finishing copy op to oid 10 low_tier 2024-09-13T11:32:27.111 INFO:tasks.rados.rados.0.smithi204.stdout:59: done (15 left) 2024-09-13T11:32:27.112 INFO:tasks.rados.rados.0.smithi204.stdout:69: copy oid 20 target oid 20 low_tier 2024-09-13T11:32:27.112 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 20 to 20 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.112 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.116 INFO:tasks.rados.rados.0.smithi204.stdout:51: finishing copy op to oid 2 low_tier 2024-09-13T11:32:27.116 INFO:tasks.rados.rados.0.smithi204.stdout:51: done (15 left) 2024-09-13T11:32:27.116 INFO:tasks.rados.rados.0.smithi204.stdout:70: copy oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.116 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 21 to 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.116 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.117 INFO:tasks.rados.rados.0.smithi204.stdout:64: finishing copy op to oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.117 INFO:tasks.rados.rados.0.smithi204.stdout:64: done (15 left) 2024-09-13T11:32:27.117 INFO:tasks.rados.rados.0.smithi204.stdout:71: copy oid 22 target oid 22 low_tier 2024-09-13T11:32:27.117 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 22 to 22 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.117 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.119 INFO:tasks.rados.rados.0.smithi204.stdout:65: finishing copy op to oid 16 low_tier 2024-09-13T11:32:27.119 INFO:tasks.rados.rados.0.smithi204.stdout:65: done (15 left) 2024-09-13T11:32:27.119 INFO:tasks.rados.rados.0.smithi204.stdout:72: copy oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.119 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 23 to 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.120 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.131 INFO:tasks.rados.rados.0.smithi204.stdout:55: finishing copy op to oid 6 low_tier 2024-09-13T11:32:27.131 INFO:tasks.rados.rados.0.smithi204.stdout:55: done (15 left) 2024-09-13T11:32:27.132 INFO:tasks.rados.rados.0.smithi204.stdout:73: copy oid 24 target oid 24 low_tier 2024-09-13T11:32:27.132 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 24 to 24 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.132 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.133 INFO:tasks.rados.rados.0.smithi204.stdout:61: finishing copy op to oid 12 low_tier 2024-09-13T11:32:27.133 INFO:tasks.rados.rados.0.smithi204.stdout:61: done (15 left) 2024-09-13T11:32:27.133 INFO:tasks.rados.rados.0.smithi204.stdout:74: copy oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.133 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 25 to 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.133 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.134 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:27.132+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:60: finishing copy op to oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:60: done (15 left) 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:75: copy oid 26 target oid 26 low_tier 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 26 to 26 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:58: finishing copy op to oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:58: done (15 left) 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:76: copy oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 27 to 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.135 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.140 INFO:tasks.rados.rados.0.smithi204.stdout:63: finishing copy op to oid 14 low_tier 2024-09-13T11:32:27.140 INFO:tasks.rados.rados.0.smithi204.stdout:63: done (15 left) 2024-09-13T11:32:27.140 INFO:tasks.rados.rados.0.smithi204.stdout:77: copy oid 28 target oid 28 low_tier 2024-09-13T11:32:27.140 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 28 to 28 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.140 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.141 INFO:tasks.rados.rados.0.smithi204.stdout:53: finishing copy op to oid 4 low_tier 2024-09-13T11:32:27.141 INFO:tasks.rados.rados.0.smithi204.stdout:53: done (15 left) 2024-09-13T11:32:27.141 INFO:tasks.rados.rados.0.smithi204.stdout:78: copy oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.141 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 29 to 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.142 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.142 INFO:tasks.rados.rados.0.smithi204.stdout:52: finishing copy op to oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.142 INFO:tasks.rados.rados.0.smithi204.stdout:52: done (15 left) 2024-09-13T11:32:27.142 INFO:tasks.rados.rados.0.smithi204.stdout:79: copy oid 30 target oid 30 low_tier 2024-09-13T11:32:27.142 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 30 to 30 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.142 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.147 INFO:tasks.rados.rados.0.smithi204.stdout:66: finishing copy op to oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:66: done (15 left) 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:80: copy oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 31 to 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:67: finishing copy op to oid 18 low_tier 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:67: done (15 left) 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:81: copy oid 32 target oid 32 low_tier 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 32 to 32 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:68: finishing copy op to oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:68: done (15 left) 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:82: copy oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.148 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 33 to 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.149 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:62: finishing copy op to oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:62: done (15 left) 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:83: copy oid 34 target oid 34 low_tier 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 34 to 34 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:57: finishing copy op to oid 8 low_tier 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:57: done (15 left) 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:84: copy oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 35 to 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:70: finishing copy op to oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.151 INFO:tasks.rados.rados.0.smithi204.stdout:70: done (15 left) 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout:85: copy oid 36 target oid 36 low_tier 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 36 to 36 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout:73: finishing copy op to oid 24 low_tier 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout:73: done (15 left) 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout:86: copy oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 37 to 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.152 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.153 INFO:tasks.rados.rados.0.smithi204.stdout:71: finishing copy op to oid 22 low_tier 2024-09-13T11:32:27.153 INFO:tasks.rados.rados.0.smithi204.stdout:71: done (15 left) 2024-09-13T11:32:27.153 INFO:tasks.rados.rados.0.smithi204.stdout:87: copy oid 38 target oid 38 low_tier 2024-09-13T11:32:27.154 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 38 to 38 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.154 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.154 INFO:tasks.rados.rados.0.smithi204.stdout:72: finishing copy op to oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.154 INFO:tasks.rados.rados.0.smithi204.stdout:72: done (15 left) 2024-09-13T11:32:27.154 INFO:tasks.rados.rados.0.smithi204.stdout:88: copy oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.154 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 39 to 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.154 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.160 INFO:tasks.rados.rados.0.smithi204.stdout:74: finishing copy op to oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout:74: done (15 left) 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout:89: copy oid 40 target oid 40 low_tier 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 40 to 40 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout:75: finishing copy op to oid 26 low_tier 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout:75: done (15 left) 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout:90: copy oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 41 to 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.161 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.166 INFO:tasks.rados.rados.0.smithi204.stdout:69: finishing copy op to oid 20 low_tier 2024-09-13T11:32:27.166 INFO:tasks.rados.rados.0.smithi204.stdout:69: done (15 left) 2024-09-13T11:32:27.166 INFO:tasks.rados.rados.0.smithi204.stdout:91: copy oid 42 target oid 42 low_tier 2024-09-13T11:32:27.166 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 42 to 42 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.166 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.169 INFO:tasks.rados.rados.0.smithi204.stdout:77: finishing copy op to oid 28 low_tier 2024-09-13T11:32:27.169 INFO:tasks.rados.rados.0.smithi204.stdout:77: done (15 left) 2024-09-13T11:32:27.169 INFO:tasks.rados.rados.0.smithi204.stdout:92: copy oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.169 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 43 to 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.169 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.014109846000000001, 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014138576, 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:32:27.170 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0100000000000005e-07 2024-09-13T11:32:27.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164267+0000", 2024-09-13T11:32:27.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090294950000000002 2024-09-13T11:32:27.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164286+0000", 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9663000000000001e-05 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164298+0000", 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1813e-05 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164398+0000", 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9739000000000002e-05 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164430+0000", 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2458e-05 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-13T11:32:27.175 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 1 2024-09-13T11:32:27.176 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:27.177 INFO:tasks.rados.rados.0.smithi204.stdout:83: finishing copy op to oid 34 low_tier 2024-09-13T11:32:27.177 INFO:tasks.rados.rados.0.smithi204.stdout:83: done (15 left) 2024-09-13T11:32:27.177 INFO:tasks.rados.rados.0.smithi204.stdout:93: copy oid 44 target oid 44 low_tier 2024-09-13T11:32:27.177 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 44 to 44 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.178 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.180 INFO:tasks.rados.rados.0.smithi204.stdout:78: finishing copy op to oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.180 INFO:tasks.rados.rados.0.smithi204.stdout:78: done (15 left) 2024-09-13T11:32:27.180 INFO:tasks.rados.rados.0.smithi204.stdout:94: copy oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.180 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 45 to 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.180 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:82: finishing copy op to oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:82: done (15 left) 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:95: copy oid 46 target oid 46 low_tier 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 46 to 46 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:81: finishing copy op to oid 32 low_tier 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:81: done (15 left) 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:96: copy oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 47 to 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.183 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:85: finishing copy op to oid 36 low_tier 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:85: done (15 left) 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:97: copy oid 48 target oid 48 low_tier 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 48 to 48 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:84: finishing copy op to oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:84: done (15 left) 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:98: copy oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 49 to 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.186 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.194 INFO:tasks.rados.rados.0.smithi204.stdout:80: finishing copy op to oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.194 INFO:tasks.rados.rados.0.smithi204.stdout:80: done (15 left) 2024-09-13T11:32:27.194 INFO:tasks.rados.rados.0.smithi204.stdout:99: copy oid 50 target oid 50 low_tier 2024-09-13T11:32:27.194 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 50 to 50 low_tier tgt_pool_name low_tier 2024-09-13T11:32:27.194 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.195 INFO:tasks.rados.rados.0.smithi204.stdout:90: finishing copy op to oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.195 INFO:tasks.rados.rados.0.smithi204.stdout:90: done (15 left) 2024-09-13T11:32:27.195 INFO:tasks.rados.rados.0.smithi204.stdout:100: copy oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.195 INFO:tasks.rados.rados.0.smithi204.stdout:copy op oid 1 to 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-13T11:32:27.195 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.198 INFO:tasks.rados.rados.0.smithi204.stdout:76: finishing copy op to oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.198 INFO:tasks.rados.rados.0.smithi204.stdout:76: done (15 left) 2024-09-13T11:32:27.198 INFO:tasks.rados.rados.0.smithi204.stdout:101: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.198 INFO:tasks.rados.rados.0.smithi204.stdout:101: set_chunk oid 2 offset: 952320 length: 609280 target oid smithi20438004-2 low_tier offset: 952320 snap -1 2024-09-13T11:32:27.198 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.199 INFO:tasks.rados.rados.0.smithi204.stdout:89: finishing copy op to oid 40 low_tier 2024-09-13T11:32:27.199 INFO:tasks.rados.rados.0.smithi204.stdout:89: done (15 left) 2024-09-13T11:32:27.199 INFO:tasks.rados.rados.0.smithi204.stdout:102: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.199 INFO:tasks.rados.rados.0.smithi204.stdout:102: set_chunk oid 3 offset: 653312 length: 475648 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 653312 snap -1 2024-09-13T11:32:27.199 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.200 INFO:tasks.rados.rados.0.smithi204.stdout:79: finishing copy op to oid 30 low_tier 2024-09-13T11:32:27.200 INFO:tasks.rados.rados.0.smithi204.stdout:79: done (15 left) 2024-09-13T11:32:27.200 INFO:tasks.rados.rados.0.smithi204.stdout:103: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.200 INFO:tasks.rados.rados.0.smithi204.stdout:103: set_chunk oid 4 offset: 457728 length: 1001984 target oid smithi20438004-4 low_tier offset: 457728 snap -1 2024-09-13T11:32:27.200 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.202 INFO:tasks.rados.rados.0.smithi204.stdout:96: finishing copy op to oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.202 INFO:tasks.rados.rados.0.smithi204.stdout:96: done (15 left) 2024-09-13T11:32:27.202 INFO:tasks.rados.rados.0.smithi204.stdout:104: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.202 INFO:tasks.rados.rados.0.smithi204.stdout:104: set_chunk oid 5 offset: 683008 length: 1364480 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 683008 snap -1 2024-09-13T11:32:27.202 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.211 INFO:tasks.rados.rados.0.smithi204.stdout:87: finishing copy op to oid 38 low_tier 2024-09-13T11:32:27.211 INFO:tasks.rados.rados.0.smithi204.stdout:86: finishing copy op to oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.212 INFO:tasks.rados.rados.0.smithi204.stdout:86: done (15 left) 2024-09-13T11:32:27.212 INFO:tasks.rados.rados.0.smithi204.stdout:87: done (14 left) 2024-09-13T11:32:27.212 INFO:tasks.rados.rados.0.smithi204.stdout:105: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.212 INFO:tasks.rados.rados.0.smithi204.stdout:105: set_chunk oid 6 offset: 766976 length: 1152512 target oid smithi20438004-6 low_tier offset: 766976 snap -1 2024-09-13T11:32:27.212 INFO:tasks.rados.rados.0.smithi204.stdout:106: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.212 INFO:tasks.rados.rados.0.smithi204.stdout:106: set_chunk oid 7 offset: 1487360 length: 460800 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1487360 snap -1 2024-09-13T11:32:27.212 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.213 INFO:tasks.rados.rados.0.smithi204.stdout:88: finishing copy op to oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.213 INFO:tasks.rados.rados.0.smithi204.stdout:88: done (15 left) 2024-09-13T11:32:27.213 INFO:tasks.rados.rados.0.smithi204.stdout:107: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.214 INFO:tasks.rados.rados.0.smithi204.stdout:107: set_chunk oid 8 offset: 302592 length: 1200128 target oid smithi20438004-8 low_tier offset: 302592 snap -1 2024-09-13T11:32:27.214 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.214 INFO:tasks.rados.rados.0.smithi204.stdout:91: finishing copy op to oid 42 low_tier 2024-09-13T11:32:27.214 INFO:tasks.rados.rados.0.smithi204.stdout:91: done (15 left) 2024-09-13T11:32:27.214 INFO:tasks.rados.rados.0.smithi204.stdout:108: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.214 INFO:tasks.rados.rados.0.smithi204.stdout:108: set_chunk oid 9 offset: 522240 length: 241664 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 522240 snap -1 2024-09-13T11:32:27.214 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.218 INFO:tasks.rados.rados.0.smithi204.stdout:95: finishing copy op to oid 46 low_tier 2024-09-13T11:32:27.218 INFO:tasks.rados.rados.0.smithi204.stdout:95: done (15 left) 2024-09-13T11:32:27.218 INFO:tasks.rados.rados.0.smithi204.stdout:109: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.218 INFO:tasks.rados.rados.0.smithi204.stdout:109: set_chunk oid 10 offset: 638976 length: 965120 target oid smithi20438004-10 low_tier offset: 638976 snap -1 2024-09-13T11:32:27.218 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout:101: finishing set_chunk to oid 2 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout:101: done (15 left) 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout:110: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout:110: set_chunk oid 11 offset: 1625600 length: 407552 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1625600 snap -1 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout:93: finishing copy op to oid 44 low_tier 2024-09-13T11:32:27.220 INFO:tasks.rados.rados.0.smithi204.stdout:93: done (15 left) 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout:111: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout:111: set_chunk oid 12 offset: 256000 length: 1519616 target oid smithi20438004-12 low_tier offset: 256000 snap -1 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout:102: finishing set_chunk to oid 3 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout:102: done (15 left) 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout:112: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout:112: set_chunk oid 13 offset: 134144 length: 1866240 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 134144 snap -1 2024-09-13T11:32:27.221 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:94: finishing copy op to oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:94: done (15 left) 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:113: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:113: set_chunk oid 14 offset: 599040 length: 716288 target oid smithi20438004-14 low_tier offset: 599040 snap -1 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:98: finishing copy op to oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:98: done (15 left) 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:114: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.224 INFO:tasks.rados.rados.0.smithi204.stdout:114: set_chunk oid 15 offset: 326144 length: 1241088 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 326144 snap -1 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:92: finishing copy op to oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:92: done (15 left) 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:115: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:115: set_chunk oid 16 offset: 657920 length: 666112 target oid smithi20438004-16 low_tier offset: 657920 snap -1 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:103: finishing set_chunk to oid 4 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:103: done (15 left) 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:116: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:116: set_chunk oid 17 offset: 724992 length: 1302016 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 724992 snap -1 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.225 INFO:tasks.rados.rados.0.smithi204.stdout:105: finishing set_chunk to oid 6 2024-09-13T11:32:27.226 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.226 INFO:tasks.rados.rados.0.smithi204.stdout:105: done (15 left) 2024-09-13T11:32:27.226 INFO:tasks.rados.rados.0.smithi204.stdout:117: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.226 INFO:tasks.rados.rados.0.smithi204.stdout:117: set_chunk oid 18 offset: 1141760 length: 76288 target oid smithi20438004-18 low_tier offset: 1141760 snap -1 2024-09-13T11:32:27.226 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.226 INFO:tasks.rados.rados.0.smithi204.stdout:104: finishing set_chunk to oid 5 2024-09-13T11:32:27.226 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.229 INFO:tasks.rados.rados.0.smithi204.stdout:104: done (15 left) 2024-09-13T11:32:27.229 INFO:tasks.rados.rados.0.smithi204.stdout:118: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.229 INFO:tasks.rados.rados.0.smithi204.stdout:118: set_chunk oid 19 offset: 1139200 length: 815104 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1139200 snap -1 2024-09-13T11:32:27.229 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.235 INFO:tasks.rados.rados.0.smithi204.stdout:100: finishing copy op to oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.235 INFO:tasks.rados.rados.0.smithi204.stdout:100: done (15 left) 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:119: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:119: set_chunk oid 20 offset: 1048064 length: 942080 target oid smithi20438004-20 low_tier offset: 1048064 snap -1 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:97: finishing copy op to oid 48 low_tier 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:97: done (15 left) 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:120: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:120: set_chunk oid 21 offset: 181760 length: 1938944 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 181760 snap -1 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:106: finishing set_chunk to oid 7 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:106: done (15 left) 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:121: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.236 INFO:tasks.rados.rados.0.smithi204.stdout:121: set_chunk oid 22 offset: 412160 length: 1727488 target oid smithi20438004-22 low_tier offset: 412160 snap -1 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:111: finishing set_chunk to oid 12 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:111: done (15 left) 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:122: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:122: set_chunk oid 23 offset: 300544 length: 1419264 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 300544 snap -1 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:113: finishing set_chunk to oid 14 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:117: finishing set_chunk to oid 18 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:113: done (15 left) 2024-09-13T11:32:27.237 INFO:tasks.rados.rados.0.smithi204.stdout:117: done (14 left) 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:123: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:123: set_chunk oid 24 offset: 4096 length: 858112 target oid smithi20438004-24 low_tier offset: 4096 snap -1 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:124: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:124: set_chunk oid 25 offset: 959488 length: 606720 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 959488 snap -1 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:114: finishing set_chunk to oid 15 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:114: done (15 left) 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:125: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:125: set_chunk oid 26 offset: 1491456 length: 677376 target oid smithi20438004-26 low_tier offset: 1491456 snap -1 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:110: finishing set_chunk to oid 11 2024-09-13T11:32:27.238 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:107: finishing set_chunk to oid 8 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:107: done (15 left) 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:110: done (14 left) 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:126: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:126: set_chunk oid 27 offset: 1241088 length: 919040 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1241088 snap -1 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:127: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:127: set_chunk oid 28 offset: 338432 length: 912896 target oid smithi20438004-28 low_tier offset: 338432 snap -1 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:108: finishing set_chunk to oid 9 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:108: done (15 left) 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:128: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.239 INFO:tasks.rados.rados.0.smithi204.stdout:128: set_chunk oid 29 offset: 452096 length: 870912 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 452096 snap -1 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:112: finishing set_chunk to oid 13 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:112: done (15 left) 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:129: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:129: set_chunk oid 30 offset: 1047552 length: 740352 target oid smithi20438004-30 low_tier offset: 1047552 snap -1 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:116: finishing set_chunk to oid 17 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:116: done (15 left) 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:130: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:130: set_chunk oid 31 offset: 12800 length: 180224 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 12800 snap -1 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.240 INFO:tasks.rados.rados.0.smithi204.stdout:99: finishing copy op to oid 50 low_tier 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:99: done (15 left) 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:131: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:131: set_chunk oid 32 offset: 247808 length: 1672704 target oid smithi20438004-32 low_tier offset: 247808 snap -1 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:109: finishing set_chunk to oid 10 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:109: done (15 left) 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:132: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:132: set_chunk oid 33 offset: 245760 length: 1932288 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 245760 snap -1 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:122: finishing set_chunk to oid 23 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:122: done (15 left) 2024-09-13T11:32:27.241 INFO:tasks.rados.rados.0.smithi204.stdout:133: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:133: set_chunk oid 34 offset: 3584 length: 966656 target oid smithi20438004-34 low_tier offset: 3584 snap -1 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:123: finishing set_chunk to oid 24 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:123: done (15 left) 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:134: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:134: set_chunk oid 35 offset: 121856 length: 799744 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 121856 snap -1 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:118: finishing set_chunk to oid 19 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:118: done (15 left) 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:135: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout:135: set_chunk oid 36 offset: 711680 length: 910848 target oid smithi20438004-36 low_tier offset: 711680 snap -1 2024-09-13T11:32:27.242 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:120: finishing set_chunk to oid 21 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:120: done (15 left) 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:136: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:136: set_chunk oid 37 offset: 1298432 length: 1263104 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1298432 snap -1 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:119: finishing set_chunk to oid 20 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:119: done (15 left) 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:137: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout:137: set_chunk oid 38 offset: 702976 length: 34304 target oid smithi20438004-38 low_tier offset: 702976 snap -1 2024-09-13T11:32:27.243 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.244 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:27.243+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:27.313 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:27.313 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:27.313 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:27.313 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:27.313 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:27.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-13T11:32:27.356 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:27.349+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:27.379 INFO:tasks.rados.rados.0.smithi204.stdout:121: finishing set_chunk to oid 22 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:125: finishing set_chunk to oid 26 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:121: done (15 left) 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:125: done (14 left) 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:138: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:138: set_chunk oid 39 offset: 464896 length: 1615360 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 464896 snap -1 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:139: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:139: set_chunk oid 40 offset: 1066496 length: 970752 target oid smithi20438004-40 low_tier offset: 1066496 snap -1 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:124: finishing set_chunk to oid 25 2024-09-13T11:32:27.380 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:124: done (15 left) 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:140: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:140: set_chunk oid 41 offset: 73216 length: 91648 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 73216 snap -1 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:126: finishing set_chunk to oid 27 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:126: done (15 left) 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:141: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:141: set_chunk oid 42 offset: 2618368 length: 18944 target oid smithi20438004-42 low_tier offset: 2618368 snap -1 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:127: finishing set_chunk to oid 28 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:127: done (15 left) 2024-09-13T11:32:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:142: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:142: set_chunk oid 43 offset: 473088 length: 1104384 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 473088 snap -1 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:132: finishing set_chunk to oid 33 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:132: done (15 left) 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:143: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:143: set_chunk oid 44 offset: 441856 length: 2180096 target oid smithi20438004-44 low_tier offset: 441856 snap -1 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:128: finishing set_chunk to oid 29 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:128: done (15 left) 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:144: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout:144: set_chunk oid 45 offset: 291328 length: 1068032 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 291328 snap -1 2024-09-13T11:32:27.382 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:130: finishing set_chunk to oid 31 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:130: done (15 left) 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:145: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:145: set_chunk oid 46 offset: 761344 length: 635392 target oid smithi20438004-46 low_tier offset: 761344 snap -1 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:129: finishing set_chunk to oid 30 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:129: done (15 left) 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:146: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:146: set_chunk oid 47 offset: 1088512 length: 534016 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1088512 snap -1 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:139: finishing set_chunk to oid 40 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.383 INFO:tasks.rados.rados.0.smithi204.stdout:139: done (15 left) 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:147: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:147: set_chunk oid 48 offset: 1232384 length: 828928 target oid smithi20438004-48 low_tier offset: 1232384 snap -1 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:115: finishing set_chunk to oid 16 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:115: done (15 left) 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:148: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:148: set_chunk oid 49 offset: 1640960 length: 16896 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1640960 snap -1 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:133: finishing set_chunk to oid 34 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:133: done (15 left) 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:149: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.384 INFO:tasks.rados.rados.0.smithi204.stdout:149: set_chunk oid 50 offset: 806912 length: 1219072 target oid smithi20438004-50 low_tier offset: 806912 snap -1 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout:134: finishing set_chunk to oid 35 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout:134: done (15 left) 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout:150: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout:150: set_chunk oid 1 offset: 222720 length: 596992 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 222720 snap -1 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout:135: finishing set_chunk to oid 36 2024-09-13T11:32:27.385 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:135: done (15 left) 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:151: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:151: set_chunk oid 2 offset: 880640 length: 239104 target oid smithi20438004-2 low_tier offset: 880640 snap -1 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:138: finishing set_chunk to oid 39 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:138: done (15 left) 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:152: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:152: set_chunk oid 3 offset: 891904 length: 1024 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 891904 snap -1 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:151: finishing set_chunk to oid 2 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 880640 length: 239104 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.386 INFO:tasks.rados.rados.0.smithi204.stdout:151: done (15 left) 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:153: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:153: set_chunk oid 4 offset: 1538560 length: 250880 target oid smithi20438004-4 low_tier offset: 1538560 snap -1 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:152: finishing set_chunk to oid 3 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 891904 length: 1024 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:152: done (15 left) 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:154: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:154: set_chunk oid 5 offset: 899072 length: 1013248 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 899072 snap -1 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:154: finishing set_chunk to oid 5 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 899072 length: 1013248 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.387 INFO:tasks.rados.rados.0.smithi204.stdout:154: done (15 left) 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:155: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:155: set_chunk oid 6 offset: 1355264 length: 304128 target oid smithi20438004-6 low_tier offset: 1355264 snap -1 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:155: finishing set_chunk to oid 6 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1355264 length: 304128 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:155: done (15 left) 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:156: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:156: set_chunk oid 7 offset: 77824 length: 422400 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 77824 snap -1 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:136: finishing set_chunk to oid 37 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 3 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:136: done (15 left) 2024-09-13T11:32:27.388 INFO:tasks.rados.rados.0.smithi204.stdout:157: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:157: set_chunk oid 8 offset: 69120 length: 974336 target oid smithi20438004-8 low_tier offset: 69120 snap -1 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:137: finishing set_chunk to oid 38 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:137: done (15 left) 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:158: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:158: set_chunk oid 9 offset: 283648 length: 834560 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 283648 snap -1 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:157: finishing set_chunk to oid 8 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 69120 length: 974336 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:157: done (15 left) 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:159: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.389 INFO:tasks.rados.rados.0.smithi204.stdout:159: set_chunk oid 10 offset: 115712 length: 1107968 target oid smithi20438004-10 low_tier offset: 115712 snap -1 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:158: finishing set_chunk to oid 9 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 283648 length: 834560 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:158: done (15 left) 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:160: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:160: set_chunk oid 11 offset: 54784 length: 834560 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 54784 snap -1 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:159: finishing set_chunk to oid 10 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 115712 length: 1107968 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:159: done (15 left) 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:161: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout:161: set_chunk oid 12 offset: 1098752 length: 665600 target oid smithi20438004-12 low_tier offset: 1098752 snap -1 2024-09-13T11:32:27.390 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:161: finishing set_chunk to oid 12 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1098752 length: 665600 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:161: done (15 left) 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:162: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:162: set_chunk oid 13 offset: 475136 length: 681984 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 475136 snap -1 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:162: finishing set_chunk to oid 13 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 475136 length: 681984 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:162: done (15 left) 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:163: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.391 INFO:tasks.rados.rados.0.smithi204.stdout:163: set_chunk oid 14 offset: 1796096 length: 331264 target oid smithi20438004-14 low_tier offset: 1796096 snap -1 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:143: finishing set_chunk to oid 44 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:145: finishing set_chunk to oid 46 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:143: done (15 left) 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:145: done (14 left) 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:164: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:164: set_chunk oid 15 offset: 446976 length: 1054720 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 446976 snap -1 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:165: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:165: set_chunk oid 16 offset: 188928 length: 304128 target oid smithi20438004-16 low_tier offset: 188928 snap -1 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.392 INFO:tasks.rados.rados.0.smithi204.stdout:141: finishing set_chunk to oid 42 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:141: done (15 left) 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:166: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:166: set_chunk oid 17 offset: 58368 length: 2205696 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 58368 snap -1 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:142: finishing set_chunk to oid 43 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 18 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:142: done (15 left) 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:167: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:167: set_chunk oid 18 offset: 1936896 length: 11776 target oid smithi20438004-18 low_tier offset: 1936896 snap -1 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:131: finishing set_chunk to oid 32 2024-09-13T11:32:27.393 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:131: done (15 left) 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:168: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:168: set_chunk oid 19 offset: 451072 length: 695808 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 451072 snap -1 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:164: finishing set_chunk to oid 15 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 446976 length: 1054720 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:146: finishing set_chunk to oid 47 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:146: done (15 left) 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:164: done (14 left) 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:169: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.394 INFO:tasks.rados.rados.0.smithi204.stdout:169: set_chunk oid 20 offset: 1324544 length: 909312 target oid smithi20438004-20 low_tier offset: 1324544 snap -1 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:170: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:170: set_chunk oid 21 offset: 730112 length: 1513984 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 730112 snap -1 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:166: finishing set_chunk to oid 17 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 58368 length: 2205696 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:166: done (15 left) 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:171: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:171: set_chunk oid 22 offset: 1772032 length: 276992 target oid smithi20438004-22 low_tier offset: 1772032 snap -1 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:168: finishing set_chunk to oid 19 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 451072 length: 695808 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.395 INFO:tasks.rados.rados.0.smithi204.stdout:168: done (15 left) 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:172: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:172: set_chunk oid 23 offset: 559616 length: 1515008 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 559616 snap -1 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:169: finishing set_chunk to oid 20 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1324544 length: 909312 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:170: finishing set_chunk to oid 21 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 730112 length: 1513984 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:169: done (15 left) 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:170: done (14 left) 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:173: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.396 INFO:tasks.rados.rados.0.smithi204.stdout:173: set_chunk oid 24 offset: 469504 length: 1184256 target oid smithi20438004-24 low_tier offset: 469504 snap -1 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:174: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:174: set_chunk oid 25 offset: 756224 length: 1572864 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 756224 snap -1 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:171: finishing set_chunk to oid 22 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1772032 length: 276992 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:171: done (15 left) 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:175: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:175: set_chunk oid 26 offset: 656384 length: 1367040 target oid smithi20438004-26 low_tier offset: 656384 snap -1 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:172: finishing set_chunk to oid 23 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 559616 length: 1515008 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.397 INFO:tasks.rados.rados.0.smithi204.stdout:172: done (15 left) 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:176: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:176: set_chunk oid 27 offset: 335360 length: 212992 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 335360 snap -1 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:174: finishing set_chunk to oid 25 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 756224 length: 1572864 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:174: done (15 left) 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:177: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:177: set_chunk oid 28 offset: 873472 length: 1341440 target oid smithi20438004-28 low_tier offset: 873472 snap -1 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:175: finishing set_chunk to oid 26 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 656384 length: 1367040 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.398 INFO:tasks.rados.rados.0.smithi204.stdout:175: done (15 left) 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:178: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:178: set_chunk oid 29 offset: 566272 length: 623104 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 566272 snap -1 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:173: finishing set_chunk to oid 24 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 469504 length: 1184256 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:173: done (15 left) 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:179: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:179: set_chunk oid 30 offset: 196096 length: 316416 target oid smithi20438004-30 low_tier offset: 196096 snap -1 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:177: finishing set_chunk to oid 28 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 873472 length: 1341440 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.399 INFO:tasks.rados.rados.0.smithi204.stdout:177: done (15 left) 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:180: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:180: set_chunk oid 31 offset: 793088 length: 472576 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 793088 snap -1 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:153: finishing set_chunk to oid 4 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:153: done (15 left) 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:181: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:181: set_chunk oid 32 offset: 939008 length: 481280 target oid smithi20438004-32 low_tier offset: 939008 snap -1 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:178: finishing set_chunk to oid 29 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 566272 length: 623104 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:178: done (15 left) 2024-09-13T11:32:27.400 INFO:tasks.rados.rados.0.smithi204.stdout:182: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:182: set_chunk oid 33 offset: 629760 length: 1670656 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 629760 snap -1 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:181: finishing set_chunk to oid 32 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 939008 length: 481280 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:181: done (15 left) 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:183: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:183: set_chunk oid 34 offset: 1242112 length: 326144 target oid smithi20438004-34 low_tier offset: 1242112 snap -1 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:182: finishing set_chunk to oid 33 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 629760 length: 1670656 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:182: done (15 left) 2024-09-13T11:32:27.401 INFO:tasks.rados.rados.0.smithi204.stdout:184: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:184: set_chunk oid 35 offset: 886784 length: 1444864 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 886784 snap -1 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:184: finishing set_chunk to oid 35 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 886784 length: 1444864 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:184: done (15 left) 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:185: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:185: set_chunk oid 36 offset: 1338368 length: 654336 target oid smithi20438004-36 low_tier offset: 1338368 snap -1 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:149: finishing set_chunk to oid 50 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:185: finishing set_chunk to oid 36 2024-09-13T11:32:27.402 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1338368 length: 654336 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:149: done (15 left) 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:185: done (14 left) 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:186: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:186: set_chunk oid 37 offset: 1581056 length: 589824 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1581056 snap -1 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:187: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:187: set_chunk oid 38 offset: 353280 length: 1859584 target oid smithi20438004-38 low_tier offset: 353280 snap -1 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:144: finishing set_chunk to oid 45 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:150: finishing set_chunk to oid 1 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:147: finishing set_chunk to oid 48 2024-09-13T11:32:27.403 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:148: finishing set_chunk to oid 49 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 6 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:144: done (15 left) 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:147: done (14 left) 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:148: done (13 left) 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:150: done (12 left) 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:188: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:188: set_chunk oid 39 offset: 473088 length: 344576 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 473088 snap -1 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:189: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:189: set_chunk oid 40 offset: 1344512 length: 1038336 target oid smithi20438004-40 low_tier offset: 1344512 snap -1 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:189 retry set_chunk ! 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 14 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:187: finishing set_chunk to oid 38 2024-09-13T11:32:27.404 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 353280 length: 1859584 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:186: finishing set_chunk to oid 37 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1581056 length: 589824 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:188: finishing set_chunk to oid 39 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 473088 length: 344576 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:186: done (13 left) 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:187: done (12 left) 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:188: done (11 left) 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 11 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:140: finishing set_chunk to oid 41 2024-09-13T11:32:27.405 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:140: done (10 left) 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:189: finishing set_chunk to oid 40 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1344512 length: 1038336 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:189: done (9 left) 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:176: finishing set_chunk to oid 27 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:176: done (8 left) 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 8 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:156: finishing set_chunk to oid 7 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.406 INFO:tasks.rados.rados.0.smithi204.stdout:160: finishing set_chunk to oid 11 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:156: done (7 left) 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:160: done (6 left) 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:179: finishing set_chunk to oid 30 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:179: done (5 left) 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:163: finishing set_chunk to oid 14 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:163: done (4 left) 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:167: finishing set_chunk to oid 18 2024-09-13T11:32:27.407 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:167: done (3 left) 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:180: finishing set_chunk to oid 31 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:180: done (2 left) 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:165: finishing set_chunk to oid 16 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:165: done (1 left) 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:183: finishing set_chunk to oid 34 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.408 INFO:tasks.rados.rados.0.smithi204.stdout:183: done (0 left) 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:190: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:190: set_chunk oid 41 offset: 1257984 length: 799232 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1257984 snap -1 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:191: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:191: set_chunk oid 42 offset: 486400 length: 218112 target oid smithi20438004-42 low_tier offset: 486400 snap -1 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:192: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:192: set_chunk oid 43 offset: 181248 length: 1238016 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 181248 snap -1 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:193: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:193: set_chunk oid 44 offset: 179712 length: 1415168 target oid smithi20438004-44 low_tier offset: 179712 snap -1 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:194: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:194: set_chunk oid 45 offset: 447488 length: 1222656 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 447488 snap -1 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:195: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:195: set_chunk oid 46 offset: 1140224 length: 731136 target oid smithi20438004-46 low_tier offset: 1140224 snap -1 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:196: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.409 INFO:tasks.rados.rados.0.smithi204.stdout:196: set_chunk oid 47 offset: 940032 length: 322560 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 940032 snap -1 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:197: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:197: set_chunk oid 48 offset: 56320 length: 53248 target oid smithi20438004-48 low_tier offset: 56320 snap -1 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:198: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:198: set_chunk oid 49 offset: 1569280 length: 202240 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1569280 snap -1 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:199: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:199: set_chunk oid 50 offset: 1330688 length: 1491456 target oid smithi20438004-50 low_tier offset: 1330688 snap -1 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:200: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:200: set_chunk oid 1 offset: 740864 length: 462848 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 740864 snap -1 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:201: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:201: set_chunk oid 2 offset: 869376 length: 1138688 target oid smithi20438004-2 low_tier offset: 869376 snap -1 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:202: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:202: set_chunk oid 3 offset: 591360 length: 1363968 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 591360 snap -1 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:203: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.410 INFO:tasks.rados.rados.0.smithi204.stdout:203: set_chunk oid 4 offset: 886784 length: 573440 target oid smithi20438004-4 low_tier offset: 886784 snap -1 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:204: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:204: set_chunk oid 5 offset: 1706496 length: 281600 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1706496 snap -1 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:205: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:205: set_chunk oid 6 offset: 1434624 length: 303104 target oid smithi20438004-6 low_tier offset: 1434624 snap -1 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:195: finishing set_chunk to oid 46 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1140224 length: 731136 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:195: done (15 left) 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:206: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:206: set_chunk oid 7 offset: 593408 length: 1025024 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 593408 snap -1 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:198: finishing set_chunk to oid 49 2024-09-13T11:32:27.411 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1569280 length: 202240 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:198: done (15 left) 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:207: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:207: set_chunk oid 8 offset: 1061888 length: 1009664 target oid smithi20438004-8 low_tier offset: 1061888 snap -1 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:196: finishing set_chunk to oid 47 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 940032 length: 322560 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:193: finishing set_chunk to oid 44 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 179712 length: 1415168 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:204: finishing set_chunk to oid 5 2024-09-13T11:32:27.412 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1706496 length: 281600 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:201: finishing set_chunk to oid 2 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 869376 length: 1138688 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:193: done (15 left) 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:196: done (14 left) 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:201: done (13 left) 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:204: done (12 left) 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:208: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:208: set_chunk oid 9 offset: 267776 length: 442880 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 267776 snap -1 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:209: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:209: set_chunk oid 10 offset: 971264 length: 40960 target oid smithi20438004-10 low_tier offset: 971264 snap -1 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:210: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:210: set_chunk oid 11 offset: 208384 length: 34304 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 208384 snap -1 2024-09-13T11:32:27.413 INFO:tasks.rados.rados.0.smithi204.stdout:211: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:211: set_chunk oid 12 offset: 107520 length: 1268224 target oid smithi20438004-12 low_tier offset: 107520 snap -1 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:194: finishing set_chunk to oid 45 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 447488 length: 1222656 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:194: done (15 left) 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:212: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:212: set_chunk oid 13 offset: 16896 length: 129536 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 16896 snap -1 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:200: finishing set_chunk to oid 1 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 740864 length: 462848 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:199: finishing set_chunk to oid 50 2024-09-13T11:32:27.414 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1330688 length: 1491456 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:202: finishing set_chunk to oid 3 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 591360 length: 1363968 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:192: finishing set_chunk to oid 43 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 181248 length: 1238016 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:203: finishing set_chunk to oid 4 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 886784 length: 573440 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:192: done (15 left) 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:199: done (14 left) 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:200: done (13 left) 2024-09-13T11:32:27.415 INFO:tasks.rados.rados.0.smithi204.stdout:202: done (12 left) 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:203: done (11 left) 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:213: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:213: set_chunk oid 14 offset: 16896 length: 344064 target oid smithi20438004-14 low_tier offset: 16896 snap -1 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:214: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:214: set_chunk oid 15 offset: 336384 length: 1748480 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 336384 snap -1 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:215: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:215: set_chunk oid 16 offset: 1536 length: 1576448 target oid smithi20438004-16 low_tier offset: 1536 snap -1 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:216: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:216: set_chunk oid 17 offset: 735232 length: 103424 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 735232 snap -1 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:217: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout:217: set_chunk oid 18 offset: 588800 length: 617472 target oid smithi20438004-18 low_tier offset: 588800 snap -1 2024-09-13T11:32:27.416 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:205: finishing set_chunk to oid 6 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1434624 length: 303104 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:206: finishing set_chunk to oid 7 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 593408 length: 1025024 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:205: done (15 left) 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:206: done (14 left) 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:218: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:218: set_chunk oid 19 offset: 935936 length: 1366016 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 935936 snap -1 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:211: finishing set_chunk to oid 12 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 107520 length: 1268224 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.417 INFO:tasks.rados.rados.0.smithi204.stdout:207: finishing set_chunk to oid 8 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1061888 length: 1009664 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:207: done (14 left) 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:211: done (13 left) 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:219: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:219: set_chunk oid 20 offset: 1097728 length: 1209856 target oid smithi20438004-20 low_tier offset: 1097728 snap -1 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:220: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:220: set_chunk oid 21 offset: 330752 length: 1517568 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 330752 snap -1 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:221: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:221: set_chunk oid 22 offset: 595456 length: 1617408 target oid smithi20438004-22 low_tier offset: 595456 snap -1 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:210: finishing set_chunk to oid 11 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 208384 length: 34304 2024-09-13T11:32:27.418 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:210: done (15 left) 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:222: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:222: set_chunk oid 23 offset: 942592 length: 769536 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 942592 snap -1 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:208: finishing set_chunk to oid 9 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 267776 length: 442880 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:208: done (15 left) 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:223: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:223: set_chunk oid 24 offset: 43520 length: 2327552 target oid smithi20438004-24 low_tier offset: 43520 snap -1 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:212: finishing set_chunk to oid 13 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 16896 length: 129536 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.419 INFO:tasks.rados.rados.0.smithi204.stdout:212: done (15 left) 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:224: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:224: set_chunk oid 25 offset: 264704 length: 45568 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 264704 snap -1 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:218: finishing set_chunk to oid 19 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 935936 length: 1366016 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:219: finishing set_chunk to oid 20 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1097728 length: 1209856 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:218: done (15 left) 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:219: done (14 left) 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:225: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:215: finishing set_chunk to oid 16 2024-09-13T11:32:27.420 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1536 length: 1576448 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:225: set_chunk oid 26 offset: 1426944 length: 16896 target oid smithi20438004-26 low_tier offset: 1426944 snap -1 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:215: done (14 left) 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:226: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:226: set_chunk oid 27 offset: 8704 length: 366080 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 8704 snap -1 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:227: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:227: set_chunk oid 28 offset: 954880 length: 556544 target oid smithi20438004-28 low_tier offset: 954880 snap -1 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:220: finishing set_chunk to oid 21 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 330752 length: 1517568 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:220: done (15 left) 2024-09-13T11:32:27.421 INFO:tasks.rados.rados.0.smithi204.stdout:228: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:228: set_chunk oid 29 offset: 1533952 length: 354304 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1533952 snap -1 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:222: finishing set_chunk to oid 23 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 942592 length: 769536 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:222: done (15 left) 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:229: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:229: set_chunk oid 30 offset: 2359296 length: 78848 target oid smithi20438004-30 low_tier offset: 2359296 snap -1 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:223: finishing set_chunk to oid 24 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 43520 length: 2327552 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:223: done (15 left) 2024-09-13T11:32:27.422 INFO:tasks.rados.rados.0.smithi204.stdout:230: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:230: set_chunk oid 31 offset: 798208 length: 671232 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 798208 snap -1 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:214: finishing set_chunk to oid 15 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 336384 length: 1748480 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:216: finishing set_chunk to oid 17 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 735232 length: 103424 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:214: done (15 left) 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:216: done (14 left) 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:231: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:231: set_chunk oid 32 offset: 1473536 length: 712192 target oid smithi20438004-32 low_tier offset: 1473536 snap -1 2024-09-13T11:32:27.423 INFO:tasks.rados.rados.0.smithi204.stdout:232: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:232: set_chunk oid 33 offset: 1892864 length: 547840 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1892864 snap -1 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:227: finishing set_chunk to oid 28 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 954880 length: 556544 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:226: finishing set_chunk to oid 27 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 8704 length: 366080 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:221: finishing set_chunk to oid 22 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 595456 length: 1617408 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:209: finishing set_chunk to oid 10 2024-09-13T11:32:27.424 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 971264 length: 40960 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:209: done (15 left) 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:221: done (14 left) 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:226: done (13 left) 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:227: done (12 left) 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:233: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:233: set_chunk oid 34 offset: 2343936 length: 108544 target oid smithi20438004-34 low_tier offset: 2343936 snap -1 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:234: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:234: set_chunk oid 35 offset: 486400 length: 632832 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 486400 snap -1 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:235: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:235: set_chunk oid 36 offset: 960512 length: 1046016 target oid smithi20438004-36 low_tier offset: 960512 snap -1 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:236: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout:236: set_chunk oid 37 offset: 1927168 length: 141824 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1927168 snap -1 2024-09-13T11:32:27.425 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:217: finishing set_chunk to oid 18 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 588800 length: 617472 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:217: done (15 left) 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:237: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:237: set_chunk oid 38 offset: 451584 length: 1731072 target oid smithi20438004-38 low_tier offset: 451584 snap -1 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:232: finishing set_chunk to oid 33 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1892864 length: 547840 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:232: done (15 left) 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:238: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout:238: set_chunk oid 39 offset: 1491968 length: 173568 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1491968 snap -1 2024-09-13T11:32:27.426 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:235: finishing set_chunk to oid 36 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 960512 length: 1046016 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:235: done (15 left) 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:239: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:239: set_chunk oid 40 offset: 491008 length: 683008 target oid smithi20438004-40 low_tier offset: 491008 snap -1 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:231: finishing set_chunk to oid 32 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1473536 length: 712192 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:231: done (15 left) 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:239 retry set_chunk ! 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 15 2024-09-13T11:32:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:230: finishing set_chunk to oid 31 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 798208 length: 671232 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:236: finishing set_chunk to oid 37 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1927168 length: 141824 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:230: done (14 left) 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:236: done (13 left) 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 13 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:237: finishing set_chunk to oid 38 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 451584 length: 1731072 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:237: done (12 left) 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 12 2024-09-13T11:32:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:238: finishing set_chunk to oid 39 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1491968 length: 173568 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:234: finishing set_chunk to oid 35 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 486400 length: 632832 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:234: done (11 left) 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:238: done (10 left) 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:239: finishing set_chunk to oid 40 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 491008 length: 683008 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:239: done (9 left) 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:197: finishing set_chunk to oid 48 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:197: done (8 left) 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 8 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:225: finishing set_chunk to oid 26 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:225: done (7 left) 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 7 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:191: finishing set_chunk to oid 42 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:191: done (6 left) 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:224: finishing set_chunk to oid 25 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:224: done (5 left) 2024-09-13T11:32:27.430 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:213: finishing set_chunk to oid 14 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:190: finishing set_chunk to oid 41 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:190: done (4 left) 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:213: done (3 left) 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:228: finishing set_chunk to oid 29 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:228: done (2 left) 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:233: finishing set_chunk to oid 34 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:233: done (1 left) 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:229: finishing set_chunk to oid 30 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:229: done (0 left) 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:240: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:240: set_chunk oid 41 offset: 2038784 length: 314368 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2038784 snap -1 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:241: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:241: set_chunk oid 42 offset: 1169408 length: 947712 target oid smithi20438004-42 low_tier offset: 1169408 snap -1 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:242: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:242: set_chunk oid 43 offset: 1161216 length: 1473024 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1161216 snap -1 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:243: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:243: set_chunk oid 44 offset: 239104 length: 1904128 target oid smithi20438004-44 low_tier offset: 239104 snap -1 2024-09-13T11:32:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:244: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.433 INFO:tasks.rados.rados.0.smithi204.stdout:244: set_chunk oid 45 offset: 2401280 length: 157696 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2401280 snap -1 2024-09-13T11:32:27.433 INFO:tasks.rados.rados.0.smithi204.stdout:245: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.433 INFO:tasks.rados.rados.0.smithi204.stdout:245: set_chunk oid 46 offset: 1024000 length: 1554944 target oid smithi20438004-46 low_tier offset: 1024000 snap -1 2024-09-13T11:32:27.433 INFO:tasks.rados.rados.0.smithi204.stdout:246: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.433 INFO:tasks.rados.rados.0.smithi204.stdout:246: set_chunk oid 47 offset: 749568 length: 1699328 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 749568 snap -1 2024-09-13T11:32:27.433 INFO:tasks.rados.rados.0.smithi204.stdout:247: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.433 INFO:tasks.rados.rados.0.smithi204.stdout:247: set_chunk oid 48 offset: 538624 length: 1676800 target oid smithi20438004-48 low_tier offset: 538624 snap -1 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:248: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:248: set_chunk oid 49 offset: 93696 length: 2680832 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 93696 snap -1 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:249: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:249: set_chunk oid 50 offset: 1048576 length: 876032 target oid smithi20438004-50 low_tier offset: 1048576 snap -1 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:250: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:250: set_chunk oid 1 offset: 23552 length: 222720 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 23552 snap -1 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:251: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:251: set_chunk oid 2 offset: 872960 length: 76288 target oid smithi20438004-2 low_tier offset: 872960 snap -1 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:252: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:252: set_chunk oid 3 offset: 1754112 length: 215552 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1754112 snap -1 2024-09-13T11:32:27.437 INFO:tasks.rados.rados.0.smithi204.stdout:253: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:253: set_chunk oid 4 offset: 524288 length: 717312 target oid smithi20438004-4 low_tier offset: 524288 snap -1 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:254: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:254: set_chunk oid 5 offset: 828928 length: 815104 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 828928 snap -1 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:255: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:255: set_chunk oid 6 offset: 527872 length: 1542144 target oid smithi20438004-6 low_tier offset: 527872 snap -1 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:240: finishing set_chunk to oid 41 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2038784 length: 314368 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:240: done (15 left) 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:256: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.438 INFO:tasks.rados.rados.0.smithi204.stdout:256: set_chunk oid 7 offset: 82944 length: 1378816 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 82944 snap -1 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:243: finishing set_chunk to oid 44 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 239104 length: 1904128 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:246: finishing set_chunk to oid 47 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 749568 length: 1699328 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:245: finishing set_chunk to oid 46 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1024000 length: 1554944 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:242: finishing set_chunk to oid 43 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1161216 length: 1473024 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.439 INFO:tasks.rados.rados.0.smithi204.stdout:242: done (15 left) 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:243: done (14 left) 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:245: done (13 left) 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:246: done (12 left) 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:257: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:257: set_chunk oid 8 offset: 183296 length: 1054208 target oid smithi20438004-8 low_tier offset: 183296 snap -1 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:258: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:258: set_chunk oid 9 offset: 1050624 length: 425472 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1050624 snap -1 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:259: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:259: set_chunk oid 10 offset: 289792 length: 1544704 target oid smithi20438004-10 low_tier offset: 289792 snap -1 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:260: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:260: set_chunk oid 11 offset: 186880 length: 856576 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 186880 snap -1 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.440 INFO:tasks.rados.rados.0.smithi204.stdout:248: finishing set_chunk to oid 49 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 93696 length: 2680832 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:248: done (15 left) 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:261: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:261: set_chunk oid 12 offset: 1898496 length: 24064 target oid smithi20438004-12 low_tier offset: 1898496 snap -1 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:249: finishing set_chunk to oid 50 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1048576 length: 876032 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:247: finishing set_chunk to oid 48 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 538624 length: 1676800 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.441 INFO:tasks.rados.rados.0.smithi204.stdout:247: done (15 left) 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:249: done (14 left) 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:262: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:262: set_chunk oid 13 offset: 151552 length: 999424 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 151552 snap -1 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:263: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:263: set_chunk oid 14 offset: 2032640 length: 141824 target oid smithi20438004-14 low_tier offset: 2032640 snap -1 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:253: finishing set_chunk to oid 4 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 524288 length: 717312 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:254: finishing set_chunk to oid 5 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 828928 length: 815104 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:253: done (15 left) 2024-09-13T11:32:27.442 INFO:tasks.rados.rados.0.smithi204.stdout:254: done (14 left) 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:264: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:264: set_chunk oid 15 offset: 93184 length: 1413120 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 93184 snap -1 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:265: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:265: set_chunk oid 16 offset: 751616 length: 786944 target oid smithi20438004-16 low_tier offset: 751616 snap -1 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:259: finishing set_chunk to oid 10 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 289792 length: 1544704 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:256: finishing set_chunk to oid 7 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 82944 length: 1378816 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:256: done (15 left) 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:259: done (14 left) 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:266: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:266: set_chunk oid 17 offset: 1176064 length: 923136 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1176064 snap -1 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:250: finishing set_chunk to oid 1 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 23552 length: 222720 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:257: finishing set_chunk to oid 8 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 183296 length: 1054208 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:250: done (14 left) 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:257: done (13 left) 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:267: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:267: set_chunk oid 18 offset: 155648 length: 734720 target oid smithi20438004-18 low_tier offset: 155648 snap -1 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:268: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:268: set_chunk oid 19 offset: 323072 length: 948224 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 323072 snap -1 2024-09-13T11:32:27.444 INFO:tasks.rados.rados.0.smithi204.stdout:269: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:269: set_chunk oid 20 offset: 1643008 length: 552448 target oid smithi20438004-20 low_tier offset: 1643008 snap -1 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:255: finishing set_chunk to oid 6 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 527872 length: 1542144 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:255: done (15 left) 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:270: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:270: set_chunk oid 21 offset: 543232 length: 886784 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 543232 snap -1 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:262: finishing set_chunk to oid 13 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 151552 length: 999424 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:262: done (15 left) 2024-09-13T11:32:27.445 INFO:tasks.rados.rados.0.smithi204.stdout:271: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:271: set_chunk oid 22 offset: 949760 length: 1397760 target oid smithi20438004-22 low_tier offset: 949760 snap -1 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:268: finishing set_chunk to oid 19 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 323072 length: 948224 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:268: done (15 left) 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:272: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:272: set_chunk oid 23 offset: 342528 length: 1456128 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 342528 snap -1 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:264: finishing set_chunk to oid 15 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 93184 length: 1413120 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.446 INFO:tasks.rados.rados.0.smithi204.stdout:266: finishing set_chunk to oid 17 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1176064 length: 923136 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:264: done (15 left) 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:266: done (14 left) 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:273: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:273: set_chunk oid 24 offset: 130560 length: 1626624 target oid smithi20438004-24 low_tier offset: 130560 snap -1 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:274: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:269: finishing set_chunk to oid 20 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1643008 length: 552448 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:260: finishing set_chunk to oid 11 2024-09-13T11:32:27.447 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 186880 length: 856576 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:263: finishing set_chunk to oid 14 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 2032640 length: 141824 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:274: set_chunk oid 25 offset: 127488 length: 867840 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 127488 snap -1 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:260: done (15 left) 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:263: done (14 left) 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:269: done (13 left) 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:275: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:275: set_chunk oid 26 offset: 303616 length: 1571840 target oid smithi20438004-26 low_tier offset: 303616 snap -1 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:276: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:276: set_chunk oid 27 offset: 1442304 length: 424448 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1442304 snap -1 2024-09-13T11:32:27.448 INFO:tasks.rados.rados.0.smithi204.stdout:271: finishing set_chunk to oid 22 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 949760 length: 1397760 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:270: finishing set_chunk to oid 21 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 543232 length: 886784 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:272: finishing set_chunk to oid 23 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 342528 length: 1456128 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:270: done (14 left) 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:271: done (13 left) 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:272: done (12 left) 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:277: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.449 INFO:tasks.rados.rados.0.smithi204.stdout:277: set_chunk oid 28 offset: 643072 length: 493056 target oid smithi20438004-28 low_tier offset: 643072 snap -1 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:278: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:278: set_chunk oid 29 offset: 1654272 length: 579072 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1654272 snap -1 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:279: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:279: set_chunk oid 30 offset: 1101312 length: 537600 target oid smithi20438004-30 low_tier offset: 1101312 snap -1 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:280: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:280: set_chunk oid 31 offset: 1221632 length: 163328 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1221632 snap -1 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:273: finishing set_chunk to oid 24 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 130560 length: 1626624 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:265: finishing set_chunk to oid 16 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 751616 length: 786944 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:265: done (15 left) 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:273: done (14 left) 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:281: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:281: set_chunk oid 32 offset: 286720 length: 1908736 target oid smithi20438004-32 low_tier offset: 286720 snap -1 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:282: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:282: set_chunk oid 33 offset: 876032 length: 530432 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 876032 snap -1 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:274: finishing set_chunk to oid 25 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 127488 length: 867840 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:274: done (15 left) 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:283: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:276: finishing set_chunk to oid 27 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1442304 length: 424448 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.451 INFO:tasks.rados.rados.0.smithi204.stdout:278: finishing set_chunk to oid 29 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1654272 length: 579072 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:277: finishing set_chunk to oid 28 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 643072 length: 493056 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:275: finishing set_chunk to oid 26 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 303616 length: 1571840 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:279: finishing set_chunk to oid 30 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1101312 length: 537600 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:283: set_chunk oid 34 offset: 1649152 length: 41472 target oid smithi20438004-34 low_tier offset: 1649152 snap -1 2024-09-13T11:32:27.452 INFO:tasks.rados.rados.0.smithi204.stdout:275: done (15 left) 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:276: done (14 left) 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:277: done (13 left) 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:278: done (12 left) 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:279: done (11 left) 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:284: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:284: set_chunk oid 35 offset: 614400 length: 353280 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 614400 snap -1 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:285: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:285: set_chunk oid 36 offset: 441344 length: 1909760 target oid smithi20438004-36 low_tier offset: 441344 snap -1 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:286: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:286: set_chunk oid 37 offset: 1099264 length: 1093632 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1099264 snap -1 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:287: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.453 INFO:tasks.rados.rados.0.smithi204.stdout:287: set_chunk oid 38 offset: 552960 length: 2044928 target oid smithi20438004-38 low_tier offset: 552960 snap -1 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:288: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:288: set_chunk oid 39 offset: 554496 length: 1008640 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 554496 snap -1 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:280: finishing set_chunk to oid 31 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1221632 length: 163328 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:282: finishing set_chunk to oid 33 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 876032 length: 530432 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.454 INFO:tasks.rados.rados.0.smithi204.stdout:280: done (15 left) 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:282: done (14 left) 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:289: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:289: set_chunk oid 40 offset: 199168 length: 464384 target oid smithi20438004-40 low_tier offset: 199168 snap -1 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:290: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:290: set_chunk oid 41 offset: 241152 length: 1600512 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 241152 snap -1 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:281: finishing set_chunk to oid 32 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 286720 length: 1908736 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:281: done (15 left) 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:290 retry set_chunk ! 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 15 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:288: finishing set_chunk to oid 39 2024-09-13T11:32:27.455 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 554496 length: 1008640 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:288: done (14 left) 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 14 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:285: finishing set_chunk to oid 36 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 441344 length: 1909760 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:287: finishing set_chunk to oid 38 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 552960 length: 2044928 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:285: done (13 left) 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:287: done (12 left) 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 12 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:286: finishing set_chunk to oid 37 2024-09-13T11:32:27.456 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1099264 length: 1093632 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:284: finishing set_chunk to oid 35 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 614400 length: 353280 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:284: done (11 left) 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:286: done (10 left) 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:290: finishing set_chunk to oid 41 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 241152 length: 1600512 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.457 INFO:tasks.rados.rados.0.smithi204.stdout:290: done (9 left) 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:251: finishing set_chunk to oid 2 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:251: done (8 left) 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 8 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:252: finishing set_chunk to oid 3 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:252: done (7 left) 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 7 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:241: finishing set_chunk to oid 42 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:241: done (6 left) 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.458 INFO:tasks.rados.rados.0.smithi204.stdout:261: finishing set_chunk to oid 12 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:261: done (5 left) 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:244: finishing set_chunk to oid 45 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:244: done (4 left) 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:258: finishing set_chunk to oid 9 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:258: done (3 left) 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.459 INFO:tasks.rados.rados.0.smithi204.stdout:289: finishing set_chunk to oid 40 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:289: done (2 left) 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:267: finishing set_chunk to oid 18 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:267: done (1 left) 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:283: finishing set_chunk to oid 34 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:283: done (0 left) 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:291: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:291: set_chunk oid 42 offset: 120832 length: 1474048 target oid smithi20438004-42 low_tier offset: 120832 snap -1 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:292: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.460 INFO:tasks.rados.rados.0.smithi204.stdout:292: set_chunk oid 43 offset: 828928 length: 1521152 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 828928 snap -1 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:293: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:293: set_chunk oid 44 offset: 425472 length: 2204672 target oid smithi20438004-44 low_tier offset: 425472 snap -1 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:294: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:294: set_chunk oid 45 offset: 317440 length: 1300992 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 317440 snap -1 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:295: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:295: set_chunk oid 46 offset: 1347584 length: 929792 target oid smithi20438004-46 low_tier offset: 1347584 snap -1 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:296: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:296: set_chunk oid 47 offset: 1162240 length: 417792 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1162240 snap -1 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:297: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:297: set_chunk oid 48 offset: 252928 length: 1762304 target oid smithi20438004-48 low_tier offset: 252928 snap -1 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:298: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:298: set_chunk oid 49 offset: 68096 length: 2481664 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 68096 snap -1 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:299: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.461 INFO:tasks.rados.rados.0.smithi204.stdout:299: set_chunk oid 50 offset: 506880 length: 2178560 target oid smithi20438004-50 low_tier offset: 506880 snap -1 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:300: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:300: set_chunk oid 1 offset: 327680 length: 246272 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 327680 snap -1 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:301: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:292: finishing set_chunk to oid 43 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 828928 length: 1521152 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:293: finishing set_chunk to oid 44 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 425472 length: 2204672 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:301: set_chunk oid 2 offset: 406528 length: 596480 target oid smithi20438004-2 low_tier offset: 406528 snap -1 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:292: done (10 left) 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:293: done (9 left) 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:302: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.462 INFO:tasks.rados.rados.0.smithi204.stdout:302: set_chunk oid 3 offset: 234496 length: 680448 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 234496 snap -1 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:303: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:303: set_chunk oid 4 offset: 1023488 length: 35328 target oid smithi20438004-4 low_tier offset: 1023488 snap -1 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:304: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:304: set_chunk oid 5 offset: 550400 length: 1051648 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 550400 snap -1 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:305: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:305: set_chunk oid 6 offset: 62976 length: 443392 target oid smithi20438004-6 low_tier offset: 62976 snap -1 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:306: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:306: set_chunk oid 7 offset: 507904 length: 599552 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 507904 snap -1 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:307: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:307: set_chunk oid 8 offset: 187392 length: 754688 target oid smithi20438004-8 low_tier offset: 187392 snap -1 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:308: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.463 INFO:tasks.rados.rados.0.smithi204.stdout:308: set_chunk oid 9 offset: 466944 length: 298496 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 466944 snap -1 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:294: finishing set_chunk to oid 45 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 317440 length: 1300992 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:294: done (15 left) 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:309: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:309: set_chunk oid 10 offset: 461824 length: 922112 target oid smithi20438004-10 low_tier offset: 461824 snap -1 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:296: finishing set_chunk to oid 47 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1162240 length: 417792 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:291: finishing set_chunk to oid 42 2024-09-13T11:32:27.464 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 120832 length: 1474048 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:299: finishing set_chunk to oid 50 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 506880 length: 2178560 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:295: finishing set_chunk to oid 46 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1347584 length: 929792 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:291: done (15 left) 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:295: done (14 left) 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:296: done (13 left) 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:299: done (12 left) 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:310: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.465 INFO:tasks.rados.rados.0.smithi204.stdout:310: set_chunk oid 11 offset: 1533952 length: 616448 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1533952 snap -1 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:311: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:311: set_chunk oid 12 offset: 1321472 length: 444416 target oid smithi20438004-12 low_tier offset: 1321472 snap -1 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:312: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:312: set_chunk oid 13 offset: 391680 length: 1129984 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 391680 snap -1 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:313: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:313: set_chunk oid 14 offset: 279040 length: 752128 target oid smithi20438004-14 low_tier offset: 279040 snap -1 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:298: finishing set_chunk to oid 49 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 68096 length: 2481664 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:297: finishing set_chunk to oid 48 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 252928 length: 1762304 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:297: done (15 left) 2024-09-13T11:32:27.466 INFO:tasks.rados.rados.0.smithi204.stdout:298: done (14 left) 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:314: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:314: set_chunk oid 15 offset: 161280 length: 1563648 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 161280 snap -1 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:315: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:315: set_chunk oid 16 offset: 1072128 length: 461824 target oid smithi20438004-16 low_tier offset: 1072128 snap -1 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:301: finishing set_chunk to oid 2 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 406528 length: 596480 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:302: finishing set_chunk to oid 3 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 234496 length: 680448 2024-09-13T11:32:27.467 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:308: finishing set_chunk to oid 9 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 466944 length: 298496 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:309: finishing set_chunk to oid 10 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 461824 length: 922112 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:304: finishing set_chunk to oid 5 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 550400 length: 1051648 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:301: done (15 left) 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:302: done (14 left) 2024-09-13T11:32:27.468 INFO:tasks.rados.rados.0.smithi204.stdout:304: done (13 left) 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:308: done (12 left) 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:309: done (11 left) 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:316: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:300: finishing set_chunk to oid 1 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 327680 length: 246272 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:316: set_chunk oid 17 offset: 963584 length: 91648 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 963584 snap -1 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:300: done (11 left) 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:317: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:317: set_chunk oid 18 offset: 882688 length: 378368 target oid smithi20438004-18 low_tier offset: 882688 snap -1 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:318: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:318: set_chunk oid 19 offset: 1728000 length: 177152 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1728000 snap -1 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:319: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.469 INFO:tasks.rados.rados.0.smithi204.stdout:319: set_chunk oid 20 offset: 740864 length: 679424 target oid smithi20438004-20 low_tier offset: 740864 snap -1 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:320: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:320: set_chunk oid 21 offset: 592896 length: 828416 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 592896 snap -1 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:321: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:321: set_chunk oid 22 offset: 1373184 length: 517632 target oid smithi20438004-22 low_tier offset: 1373184 snap -1 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:303: finishing set_chunk to oid 4 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1023488 length: 35328 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:307: finishing set_chunk to oid 8 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 187392 length: 754688 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:310: finishing set_chunk to oid 11 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1533952 length: 616448 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.470 INFO:tasks.rados.rados.0.smithi204.stdout:311: finishing set_chunk to oid 12 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1321472 length: 444416 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:312: finishing set_chunk to oid 13 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 391680 length: 1129984 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:313: finishing set_chunk to oid 14 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 279040 length: 752128 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:303: done (15 left) 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:307: done (14 left) 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:310: done (13 left) 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:311: done (12 left) 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:312: done (11 left) 2024-09-13T11:32:27.471 INFO:tasks.rados.rados.0.smithi204.stdout:313: done (10 left) 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:322: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:322: set_chunk oid 23 offset: 1375744 length: 144384 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1375744 snap -1 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:323: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:323: set_chunk oid 24 offset: 221184 length: 1417216 target oid smithi20438004-24 low_tier offset: 221184 snap -1 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:324: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:324: set_chunk oid 25 offset: 1186304 length: 617984 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1186304 snap -1 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:314: finishing set_chunk to oid 15 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 161280 length: 1563648 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:315: finishing set_chunk to oid 16 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1072128 length: 461824 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.472 INFO:tasks.rados.rados.0.smithi204.stdout:314: done (12 left) 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:315: done (11 left) 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:325: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:325: set_chunk oid 26 offset: 259584 length: 2065408 target oid smithi20438004-26 low_tier offset: 259584 snap -1 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:326: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:326: set_chunk oid 27 offset: 93184 length: 815616 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 93184 snap -1 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:327: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:327: set_chunk oid 28 offset: 235008 length: 284672 target oid smithi20438004-28 low_tier offset: 235008 snap -1 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:328: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:328: set_chunk oid 29 offset: 340992 length: 921600 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 340992 snap -1 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:329: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:329: set_chunk oid 30 offset: 754176 length: 1109504 target oid smithi20438004-30 low_tier offset: 754176 snap -1 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:317: finishing set_chunk to oid 18 2024-09-13T11:32:27.473 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 882688 length: 378368 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:316: finishing set_chunk to oid 17 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 963584 length: 91648 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:318: finishing set_chunk to oid 19 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1728000 length: 177152 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:321: finishing set_chunk to oid 22 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1373184 length: 517632 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:320: finishing set_chunk to oid 21 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 592896 length: 828416 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.474 INFO:tasks.rados.rados.0.smithi204.stdout:322: finishing set_chunk to oid 23 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1375744 length: 144384 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:319: finishing set_chunk to oid 20 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 740864 length: 679424 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:316: done (15 left) 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:317: done (14 left) 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:318: done (13 left) 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:319: done (12 left) 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:320: done (11 left) 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:321: done (10 left) 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:322: done (9 left) 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:330: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.475 INFO:tasks.rados.rados.0.smithi204.stdout:330: set_chunk oid 31 offset: 451072 length: 341504 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 451072 snap -1 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:331: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:323: finishing set_chunk to oid 24 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 221184 length: 1417216 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:331: set_chunk oid 32 offset: 1033728 length: 1186304 target oid smithi20438004-32 low_tier offset: 1033728 snap -1 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:324: finishing set_chunk to oid 25 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1186304 length: 617984 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:323: done (10 left) 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:324: done (9 left) 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:332: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:332: set_chunk oid 33 offset: 161792 length: 2164736 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 161792 snap -1 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:333: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.476 INFO:tasks.rados.rados.0.smithi204.stdout:333: set_chunk oid 34 offset: 18432 length: 509440 target oid smithi20438004-34 low_tier offset: 18432 snap -1 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:334: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:327: finishing set_chunk to oid 28 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 235008 length: 284672 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:328: finishing set_chunk to oid 29 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 340992 length: 921600 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:334: set_chunk oid 35 offset: 758272 length: 141312 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 758272 snap -1 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:327: done (11 left) 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:328: done (10 left) 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:335: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.477 INFO:tasks.rados.rados.0.smithi204.stdout:335: set_chunk oid 36 offset: 37376 length: 552448 target oid smithi20438004-36 low_tier offset: 37376 snap -1 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:336: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:336: set_chunk oid 37 offset: 925184 length: 1170944 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 925184 snap -1 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:337: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:337: set_chunk oid 38 offset: 2028544 length: 542720 target oid smithi20438004-38 low_tier offset: 2028544 snap -1 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:338: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:338: set_chunk oid 39 offset: 1809920 length: 539648 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1809920 snap -1 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:339: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:326: finishing set_chunk to oid 27 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 93184 length: 815616 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.478 INFO:tasks.rados.rados.0.smithi204.stdout:339: set_chunk oid 40 offset: 461824 length: 521728 target oid smithi20438004-40 low_tier offset: 461824 snap -1 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:325: finishing set_chunk to oid 26 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 259584 length: 2065408 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:329: finishing set_chunk to oid 30 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 754176 length: 1109504 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:325: done (14 left) 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:326: done (13 left) 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:329: done (12 left) 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:340: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:340: set_chunk oid 41 offset: 1179648 length: 1456128 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1179648 snap -1 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:341: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:341: set_chunk oid 42 offset: 377856 length: 656384 target oid smithi20438004-42 low_tier offset: 377856 snap -1 2024-09-13T11:32:27.479 INFO:tasks.rados.rados.0.smithi204.stdout:342: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:342: set_chunk oid 43 offset: 1152512 length: 519168 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1152512 snap -1 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:343: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:343: set_chunk oid 44 offset: 550400 length: 2133504 target oid smithi20438004-44 low_tier offset: 550400 snap -1 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:333: finishing set_chunk to oid 34 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 18432 length: 509440 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:331: finishing set_chunk to oid 32 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1033728 length: 1186304 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.480 INFO:tasks.rados.rados.0.smithi204.stdout:332: finishing set_chunk to oid 33 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 161792 length: 2164736 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:331: done (15 left) 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:332: done (14 left) 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:333: done (13 left) 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:344: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:344: set_chunk oid 45 offset: 1400832 length: 74240 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1400832 snap -1 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:345: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:345: set_chunk oid 46 offset: 2449408 length: 270336 target oid smithi20438004-46 low_tier offset: 2449408 snap -1 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:346: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.481 INFO:tasks.rados.rados.0.smithi204.stdout:346: set_chunk oid 47 offset: 1470464 length: 1148928 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1470464 snap -1 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:338: finishing set_chunk to oid 39 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1809920 length: 539648 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:338: done (15 left) 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:347: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:347: set_chunk oid 48 offset: 1920512 length: 637440 target oid smithi20438004-48 low_tier offset: 1920512 snap -1 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:334: finishing set_chunk to oid 35 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 758272 length: 141312 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.482 INFO:tasks.rados.rados.0.smithi204.stdout:336: finishing set_chunk to oid 37 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 925184 length: 1170944 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:334: done (15 left) 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:336: done (14 left) 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:348: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:348: set_chunk oid 49 offset: 753152 length: 70144 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 753152 snap -1 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:349: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:349: set_chunk oid 50 offset: 1200128 length: 443392 target oid smithi20438004-50 low_tier offset: 1200128 snap -1 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:339: finishing set_chunk to oid 40 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 461824 length: 521728 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:340: finishing set_chunk to oid 41 2024-09-13T11:32:27.483 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1179648 length: 1456128 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:341: finishing set_chunk to oid 42 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 377856 length: 656384 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:343: finishing set_chunk to oid 44 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 550400 length: 2133504 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:339: done (15 left) 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:340: done (14 left) 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:341: done (13 left) 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:343: done (12 left) 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:350: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.484 INFO:tasks.rados.rados.0.smithi204.stdout:350: set_chunk oid 1 offset: 236032 length: 22528 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 236032 snap -1 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:351: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:351: set_chunk oid 2 offset: 1166336 length: 685568 target oid smithi20438004-2 low_tier offset: 1166336 snap -1 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:352: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:352: set_chunk oid 3 offset: 295424 length: 1627648 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 295424 snap -1 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:353: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:353: set_chunk oid 4 offset: 1014272 length: 577024 target oid smithi20438004-4 low_tier offset: 1014272 snap -1 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:342: finishing set_chunk to oid 43 2024-09-13T11:32:27.485 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1152512 length: 519168 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:346: finishing set_chunk to oid 47 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1470464 length: 1148928 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:342: done (15 left) 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:346: done (14 left) 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:354: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:354: set_chunk oid 5 offset: 890880 length: 599552 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 890880 snap -1 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:354 retry set_chunk ! 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 15 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:349: finishing set_chunk to oid 50 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1200128 length: 443392 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.486 INFO:tasks.rados.rados.0.smithi204.stdout:349: done (14 left) 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 14 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:347: finishing set_chunk to oid 48 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1920512 length: 637440 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:347: done (13 left) 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 13 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:352: finishing set_chunk to oid 3 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 295424 length: 1627648 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:351: finishing set_chunk to oid 2 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1166336 length: 685568 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.487 INFO:tasks.rados.rados.0.smithi204.stdout:351: done (12 left) 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:352: done (11 left) 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 11 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:354: finishing set_chunk to oid 5 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 890880 length: 599552 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:354: done (10 left) 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:350: finishing set_chunk to oid 1 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 236032 length: 22528 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:350: done (9 left) 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.488 INFO:tasks.rados.rados.0.smithi204.stdout:353: finishing set_chunk to oid 4 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1014272 length: 577024 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:353: done (8 left) 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 8 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:305: finishing set_chunk to oid 6 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:305: done (7 left) 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 7 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:335: finishing set_chunk to oid 36 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:335: done (6 left) 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:306: finishing set_chunk to oid 7 2024-09-13T11:32:27.489 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:306: done (5 left) 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:330: finishing set_chunk to oid 31 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:330: done (4 left) 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:344: finishing set_chunk to oid 45 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 9 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:344: done (3 left) 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:337: finishing set_chunk to oid 38 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout:337: done (2 left) 2024-09-13T11:32:27.490 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:345: finishing set_chunk to oid 46 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:345: done (1 left) 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:348: finishing set_chunk to oid 49 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 6 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:348: done (0 left) 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:355: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:355: set_chunk oid 6 offset: 295424 length: 195072 target oid smithi20438004-6 low_tier offset: 295424 snap -1 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:356: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:356: set_chunk oid 7 offset: 1481216 length: 588288 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1481216 snap -1 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:357: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.491 INFO:tasks.rados.rados.0.smithi204.stdout:357: set_chunk oid 8 offset: 1917952 length: 178688 target oid smithi20438004-8 low_tier offset: 1917952 snap -1 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:358: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:358: set_chunk oid 9 offset: 942080 length: 693248 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 942080 snap -1 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:359: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:359: set_chunk oid 10 offset: 1149952 length: 832000 target oid smithi20438004-10 low_tier offset: 1149952 snap -1 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:360: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:360: set_chunk oid 11 offset: 602112 length: 602112 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 602112 snap -1 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:361: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:361: set_chunk oid 12 offset: 449536 length: 1188352 target oid smithi20438004-12 low_tier offset: 449536 snap -1 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:362: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:362: set_chunk oid 13 offset: 99328 length: 1723904 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 99328 snap -1 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:363: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:363: set_chunk oid 14 offset: 773120 length: 502272 target oid smithi20438004-14 low_tier offset: 773120 snap -1 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:364: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.492 INFO:tasks.rados.rados.0.smithi204.stdout:364: set_chunk oid 15 offset: 272384 length: 150016 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 272384 snap -1 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:365: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:365: set_chunk oid 16 offset: 881664 length: 1142272 target oid smithi20438004-16 low_tier offset: 881664 snap -1 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:366: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:366: set_chunk oid 17 offset: 1139200 length: 358400 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1139200 snap -1 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:367: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:367: set_chunk oid 18 offset: 643072 length: 346624 target oid smithi20438004-18 low_tier offset: 643072 snap -1 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:368: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:368: set_chunk oid 19 offset: 47616 length: 96256 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 47616 snap -1 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:369: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:369: set_chunk oid 20 offset: 1048064 length: 829952 target oid smithi20438004-20 low_tier offset: 1048064 snap -1 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:370: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout:370: set_chunk oid 21 offset: 1391616 length: 870912 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1391616 snap -1 2024-09-13T11:32:27.493 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:360: finishing set_chunk to oid 11 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 602112 length: 602112 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:360: done (15 left) 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:371: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:371: set_chunk oid 22 offset: 1578496 length: 83456 target oid smithi20438004-22 low_tier offset: 1578496 snap -1 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:361: finishing set_chunk to oid 12 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 449536 length: 1188352 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:355: finishing set_chunk to oid 6 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 295424 length: 195072 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.494 INFO:tasks.rados.rados.0.smithi204.stdout:355: done (15 left) 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:361: done (14 left) 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:372: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:372: set_chunk oid 23 offset: 1123840 length: 277504 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1123840 snap -1 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:373: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:373: set_chunk oid 24 offset: 333312 length: 1061888 target oid smithi20438004-24 low_tier offset: 333312 snap -1 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:363: finishing set_chunk to oid 14 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 773120 length: 502272 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:363: done (15 left) 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:374: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:374: set_chunk oid 25 offset: 623104 length: 1359872 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 623104 snap -1 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:369: finishing set_chunk to oid 20 2024-09-13T11:32:27.495 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1048064 length: 829952 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:369: done (15 left) 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:375: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:375: set_chunk oid 26 offset: 915456 length: 800768 target oid smithi20438004-26 low_tier offset: 915456 snap -1 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:370: finishing set_chunk to oid 21 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1391616 length: 870912 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:370: done (15 left) 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:376: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:376: set_chunk oid 27 offset: 117760 length: 278016 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 117760 snap -1 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:373: finishing set_chunk to oid 24 2024-09-13T11:32:27.496 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 333312 length: 1061888 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:373: done (15 left) 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:377: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:377: set_chunk oid 28 offset: 697856 length: 651264 target oid smithi20438004-28 low_tier offset: 697856 snap -1 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:372: finishing set_chunk to oid 23 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1123840 length: 277504 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:372: done (15 left) 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:378: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:378: set_chunk oid 29 offset: 1056768 length: 117248 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1056768 snap -1 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.497 INFO:tasks.rados.rados.0.smithi204.stdout:374: finishing set_chunk to oid 25 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 623104 length: 1359872 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:356: finishing set_chunk to oid 7 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1481216 length: 588288 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:356: done (15 left) 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:374: done (14 left) 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:379: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:379: set_chunk oid 30 offset: 658944 length: 527360 target oid smithi20438004-30 low_tier offset: 658944 snap -1 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:380: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:380: set_chunk oid 31 offset: 1121792 length: 535552 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1121792 snap -1 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:358: finishing set_chunk to oid 9 2024-09-13T11:32:27.498 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 942080 length: 693248 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:358: done (15 left) 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:381: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:381: set_chunk oid 32 offset: 2081280 length: 279552 target oid smithi20438004-32 low_tier offset: 2081280 snap -1 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:375: finishing set_chunk to oid 26 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 915456 length: 800768 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:359: finishing set_chunk to oid 10 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1149952 length: 832000 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:364: finishing set_chunk to oid 15 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 272384 length: 150016 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.499 INFO:tasks.rados.rados.0.smithi204.stdout:362: finishing set_chunk to oid 13 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 99328 length: 1723904 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:365: finishing set_chunk to oid 16 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 881664 length: 1142272 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:376: finishing set_chunk to oid 27 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 117760 length: 278016 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:377: finishing set_chunk to oid 28 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 697856 length: 651264 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:359: done (15 left) 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:362: done (14 left) 2024-09-13T11:32:27.500 INFO:tasks.rados.rados.0.smithi204.stdout:364: done (13 left) 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:365: done (12 left) 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:375: done (11 left) 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:376: done (10 left) 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:377: done (9 left) 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:382: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:382: set_chunk oid 33 offset: 1871872 length: 583168 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1871872 snap -1 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:383: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:383: set_chunk oid 34 offset: 1260032 length: 483840 target oid smithi20438004-34 low_tier offset: 1260032 snap -1 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:384: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:367: finishing set_chunk to oid 18 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 643072 length: 346624 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:366: finishing set_chunk to oid 17 2024-09-13T11:32:27.501 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1139200 length: 358400 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:384: set_chunk oid 35 offset: 365056 length: 1235968 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 365056 snap -1 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:366: done (11 left) 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:367: done (10 left) 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:385: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:385: set_chunk oid 36 offset: 100864 length: 2169856 target oid smithi20438004-36 low_tier offset: 100864 snap -1 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:386: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:386: set_chunk oid 37 offset: 82432 length: 72704 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 82432 snap -1 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:379: finishing set_chunk to oid 30 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 658944 length: 527360 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:378: finishing set_chunk to oid 29 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1056768 length: 117248 2024-09-13T11:32:27.502 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:371: finishing set_chunk to oid 22 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1578496 length: 83456 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:371: done (11 left) 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:378: done (10 left) 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:379: done (9 left) 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:387: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:387: set_chunk oid 38 offset: 1502720 length: 468480 target oid smithi20438004-38 low_tier offset: 1502720 snap -1 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:388: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:388: set_chunk oid 39 offset: 882688 length: 136704 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 882688 snap -1 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:389: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:389: set_chunk oid 40 offset: 1753600 length: 194560 target oid smithi20438004-40 low_tier offset: 1753600 snap -1 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:390: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.503 INFO:tasks.rados.rados.0.smithi204.stdout:390: set_chunk oid 41 offset: 204800 length: 1751552 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 204800 snap -1 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:391: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:391: set_chunk oid 42 offset: 529920 length: 567808 target oid smithi20438004-42 low_tier offset: 529920 snap -1 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:392: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:392: set_chunk oid 43 offset: 418304 length: 1729024 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 418304 snap -1 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:393: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:393: set_chunk oid 44 offset: 1321472 length: 1116672 target oid smithi20438004-44 low_tier offset: 1321472 snap -1 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:382: finishing set_chunk to oid 33 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1871872 length: 583168 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:385: finishing set_chunk to oid 36 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 100864 length: 2169856 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.504 INFO:tasks.rados.rados.0.smithi204.stdout:382: done (15 left) 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:385: done (14 left) 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:394: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:394: set_chunk oid 45 offset: 2028544 length: 378368 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2028544 snap -1 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:388: finishing set_chunk to oid 39 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 882688 length: 136704 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:388: done (14 left) 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:395: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:395: set_chunk oid 46 offset: 700928 length: 1398784 target oid smithi20438004-46 low_tier offset: 700928 snap -1 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:396: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:396: set_chunk oid 47 offset: 1203712 length: 1326592 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1203712 snap -1 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.505 INFO:tasks.rados.rados.0.smithi204.stdout:391: finishing set_chunk to oid 42 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 529920 length: 567808 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:391: done (15 left) 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:397: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:397: set_chunk oid 48 offset: 53760 length: 815104 target oid smithi20438004-48 low_tier offset: 53760 snap -1 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:395: finishing set_chunk to oid 46 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 700928 length: 1398784 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:395: done (15 left) 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:398: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:398: set_chunk oid 49 offset: 803840 length: 509440 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 803840 snap -1 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.506 INFO:tasks.rados.rados.0.smithi204.stdout:380: finishing set_chunk to oid 31 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1121792 length: 535552 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:383: finishing set_chunk to oid 34 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1260032 length: 483840 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:384: finishing set_chunk to oid 35 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 365056 length: 1235968 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:380: done (15 left) 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:383: done (14 left) 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:384: done (13 left) 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:399: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.507 INFO:tasks.rados.rados.0.smithi204.stdout:399: set_chunk oid 50 offset: 551936 length: 896000 target oid smithi20438004-50 low_tier offset: 551936 snap -1 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout:400: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout:400: set_chunk oid 1 offset: 1142272 length: 344576 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1142272 snap -1 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout:401: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout:401: set_chunk oid 2 offset: 1291264 length: 135168 target oid smithi20438004-2 low_tier offset: 1291264 snap -1 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout:389: finishing set_chunk to oid 40 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1753600 length: 194560 2024-09-13T11:32:27.508 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:389: done (15 left) 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:402: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:402: set_chunk oid 3 offset: 488960 length: 155648 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 488960 snap -1 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:397: finishing set_chunk to oid 48 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 53760 length: 815104 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:390: finishing set_chunk to oid 41 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 204800 length: 1751552 2024-09-13T11:32:27.509 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:394: finishing set_chunk to oid 45 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2028544 length: 378368 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:393: finishing set_chunk to oid 44 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1321472 length: 1116672 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:390: done (15 left) 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:393: done (14 left) 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:394: done (13 left) 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:397: done (12 left) 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:403: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.510 INFO:tasks.rados.rados.0.smithi204.stdout:403: set_chunk oid 4 offset: 958976 length: 334848 target oid smithi20438004-4 low_tier offset: 958976 snap -1 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:404: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:404: set_chunk oid 5 offset: 1043456 length: 345600 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1043456 snap -1 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:405: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:405: set_chunk oid 6 offset: 313344 length: 1325056 target oid smithi20438004-6 low_tier offset: 313344 snap -1 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:406: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:406: set_chunk oid 7 offset: 1287168 length: 663040 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1287168 snap -1 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:396: finishing set_chunk to oid 47 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1203712 length: 1326592 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:392: finishing set_chunk to oid 43 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 418304 length: 1729024 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.511 INFO:tasks.rados.rados.0.smithi204.stdout:398: finishing set_chunk to oid 49 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 803840 length: 509440 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:401: finishing set_chunk to oid 2 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1291264 length: 135168 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:392: done (15 left) 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:396: done (14 left) 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:398: done (13 left) 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:401: done (12 left) 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:406 retry set_chunk ! 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 12 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:403: finishing set_chunk to oid 4 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 958976 length: 334848 2024-09-13T11:32:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:403: done (11 left) 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 11 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:405: finishing set_chunk to oid 6 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 313344 length: 1325056 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:405: done (10 left) 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:404: finishing set_chunk to oid 5 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1043456 length: 345600 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:404: done (9 left) 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:399: finishing set_chunk to oid 50 2024-09-13T11:32:27.513 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 551936 length: 896000 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:399: done (8 left) 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 8 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:406: finishing set_chunk to oid 7 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1287168 length: 663040 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:406: done (7 left) 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 7 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:368: finishing set_chunk to oid 19 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:368: done (6 left) 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:381: finishing set_chunk to oid 32 2024-09-13T11:32:27.514 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:381: done (5 left) 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:357: finishing set_chunk to oid 8 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:357: done (4 left) 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:387: finishing set_chunk to oid 38 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:387: done (3 left) 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:402: finishing set_chunk to oid 3 2024-09-13T11:32:27.515 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:402: done (2 left) 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:386: finishing set_chunk to oid 37 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 3 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:386: done (1 left) 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:400: finishing set_chunk to oid 1 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:400: done (0 left) 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:407: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:407: set_chunk oid 8 offset: 922624 length: 518656 target oid smithi20438004-8 low_tier offset: 922624 snap -1 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:408: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.516 INFO:tasks.rados.rados.0.smithi204.stdout:408: set_chunk oid 9 offset: 427520 length: 1518080 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 427520 snap -1 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:409: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:409: set_chunk oid 10 offset: 1146880 length: 121344 target oid smithi20438004-10 low_tier offset: 1146880 snap -1 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:410: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:410: set_chunk oid 11 offset: 970752 length: 459264 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 970752 snap -1 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:411: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:411: set_chunk oid 12 offset: 148992 length: 1720320 target oid smithi20438004-12 low_tier offset: 148992 snap -1 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:412: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:412: set_chunk oid 13 offset: 182784 length: 1899520 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 182784 snap -1 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:413: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:413: set_chunk oid 14 offset: 79360 length: 437248 target oid smithi20438004-14 low_tier offset: 79360 snap -1 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:414: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:414: set_chunk oid 15 offset: 844800 length: 462848 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 844800 snap -1 2024-09-13T11:32:27.517 INFO:tasks.rados.rados.0.smithi204.stdout:415: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:415: set_chunk oid 16 offset: 16896 length: 84480 target oid smithi20438004-16 low_tier offset: 16896 snap -1 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:416: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:416: set_chunk oid 17 offset: 333824 length: 1399808 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 333824 snap -1 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:417: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:417: set_chunk oid 18 offset: 1678848 length: 489472 target oid smithi20438004-18 low_tier offset: 1678848 snap -1 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:418: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:418: set_chunk oid 19 offset: 478208 length: 463872 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 478208 snap -1 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:419: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:419: set_chunk oid 20 offset: 128512 length: 1799168 target oid smithi20438004-20 low_tier offset: 128512 snap -1 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:420: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:420: set_chunk oid 21 offset: 1389568 length: 252416 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1389568 snap -1 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:421: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:421: set_chunk oid 22 offset: 906752 length: 592384 target oid smithi20438004-22 low_tier offset: 906752 snap -1 2024-09-13T11:32:27.518 INFO:tasks.rados.rados.0.smithi204.stdout:422: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:422: set_chunk oid 23 offset: 307712 length: 1848832 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 307712 snap -1 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:411: finishing set_chunk to oid 12 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 148992 length: 1720320 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:411: done (15 left) 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:423: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:423: set_chunk oid 24 offset: 1955328 length: 395264 target oid smithi20438004-24 low_tier offset: 1955328 snap -1 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:420: finishing set_chunk to oid 21 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1389568 length: 252416 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.519 INFO:tasks.rados.rados.0.smithi204.stdout:413: finishing set_chunk to oid 14 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 79360 length: 437248 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:413: done (15 left) 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:420: done (14 left) 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:424: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:424: set_chunk oid 25 offset: 877056 length: 410624 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 877056 snap -1 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:425: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:425: set_chunk oid 26 offset: 576000 length: 62976 target oid smithi20438004-26 low_tier offset: 576000 snap -1 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:419: finishing set_chunk to oid 20 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 128512 length: 1799168 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:419: done (15 left) 2024-09-13T11:32:27.520 INFO:tasks.rados.rados.0.smithi204.stdout:426: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:426: set_chunk oid 27 offset: 1213440 length: 132096 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1213440 snap -1 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:422: finishing set_chunk to oid 23 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 307712 length: 1848832 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:422: done (15 left) 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:427: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:427: set_chunk oid 28 offset: 1282560 length: 547840 target oid smithi20438004-28 low_tier offset: 1282560 snap -1 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:424: finishing set_chunk to oid 25 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 877056 length: 410624 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:424: done (15 left) 2024-09-13T11:32:27.521 INFO:tasks.rados.rados.0.smithi204.stdout:428: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:428: set_chunk oid 29 offset: 1902080 length: 274944 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1902080 snap -1 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:426: finishing set_chunk to oid 27 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1213440 length: 132096 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:426: done (15 left) 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:429: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:429: set_chunk oid 30 offset: 1723904 length: 565760 target oid smithi20438004-30 low_tier offset: 1723904 snap -1 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:429: finishing set_chunk to oid 30 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1723904 length: 565760 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:429: done (15 left) 2024-09-13T11:32:27.522 INFO:tasks.rados.rados.0.smithi204.stdout:430: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:430: set_chunk oid 31 offset: 1752064 length: 485888 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1752064 snap -1 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:407: finishing set_chunk to oid 8 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 922624 length: 518656 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:407: done (15 left) 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:431: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:431: set_chunk oid 32 offset: 934400 length: 18432 target oid smithi20438004-32 low_tier offset: 934400 snap -1 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:408: finishing set_chunk to oid 9 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 427520 length: 1518080 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:408: done (15 left) 2024-09-13T11:32:27.523 INFO:tasks.rados.rados.0.smithi204.stdout:432: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:432: set_chunk oid 33 offset: 112640 length: 801792 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 112640 snap -1 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:409: finishing set_chunk to oid 10 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1146880 length: 121344 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:414: finishing set_chunk to oid 15 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 844800 length: 462848 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:416: finishing set_chunk to oid 17 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 333824 length: 1399808 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:409: done (15 left) 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:414: done (14 left) 2024-09-13T11:32:27.524 INFO:tasks.rados.rados.0.smithi204.stdout:416: done (13 left) 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:433: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:417: finishing set_chunk to oid 18 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1678848 length: 489472 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:433: set_chunk oid 34 offset: 2448384 length: 102912 target oid smithi20438004-34 low_tier offset: 2448384 snap -1 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:417: done (13 left) 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:434: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:434: set_chunk oid 35 offset: 522240 length: 1709056 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 522240 snap -1 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:435: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:435: set_chunk oid 36 offset: 1111552 length: 357376 target oid smithi20438004-36 low_tier offset: 1111552 snap -1 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:436: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout:436: set_chunk oid 37 offset: 394752 length: 300032 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 394752 snap -1 2024-09-13T11:32:27.525 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:421: finishing set_chunk to oid 22 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 906752 length: 592384 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:421: done (15 left) 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:437: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:437: set_chunk oid 38 offset: 148480 length: 1411584 target oid smithi20438004-38 low_tier offset: 148480 snap -1 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:431: finishing set_chunk to oid 32 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 934400 length: 18432 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:432: finishing set_chunk to oid 33 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 112640 length: 801792 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.526 INFO:tasks.rados.rados.0.smithi204.stdout:431: done (15 left) 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:432: done (14 left) 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:438: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:438: set_chunk oid 39 offset: 1868800 length: 390656 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1868800 snap -1 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:439: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:439: set_chunk oid 40 offset: 1775104 length: 218112 target oid smithi20438004-40 low_tier offset: 1775104 snap -1 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:435: finishing set_chunk to oid 36 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1111552 length: 357376 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:435: done (15 left) 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:440: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:440: set_chunk oid 41 offset: 2207744 length: 436736 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2207744 snap -1 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.527 INFO:tasks.rados.rados.0.smithi204.stdout:438: finishing set_chunk to oid 39 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1868800 length: 390656 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:438: done (15 left) 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:441: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:441: set_chunk oid 42 offset: 1045504 length: 881664 target oid smithi20438004-42 low_tier offset: 1045504 snap -1 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:437: finishing set_chunk to oid 38 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 148480 length: 1411584 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:437: done (15 left) 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:442: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout:442: set_chunk oid 43 offset: 1368064 length: 1139712 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1368064 snap -1 2024-09-13T11:32:27.528 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:441: finishing set_chunk to oid 42 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1045504 length: 881664 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:441: done (15 left) 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:443: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:443: set_chunk oid 44 offset: 851968 length: 51712 target oid smithi20438004-44 low_tier offset: 851968 snap -1 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:433: finishing set_chunk to oid 34 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 2448384 length: 102912 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:433: done (15 left) 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:444: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout:444: set_chunk oid 45 offset: 503296 length: 1669120 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 503296 snap -1 2024-09-13T11:32:27.529 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:434: finishing set_chunk to oid 35 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 522240 length: 1709056 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:434: done (15 left) 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:445: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:445: set_chunk oid 46 offset: 109568 length: 1573376 target oid smithi20438004-46 low_tier offset: 109568 snap -1 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:439: finishing set_chunk to oid 40 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1775104 length: 218112 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:439: done (15 left) 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:446: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout:446: set_chunk oid 47 offset: 2359296 length: 164864 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2359296 snap -1 2024-09-13T11:32:27.530 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:443: finishing set_chunk to oid 44 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 851968 length: 51712 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:443: done (15 left) 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:447: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:447: set_chunk oid 48 offset: 342016 length: 695296 target oid smithi20438004-48 low_tier offset: 342016 snap -1 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:445: finishing set_chunk to oid 46 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 109568 length: 1573376 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:445: done (15 left) 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:448: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:448: set_chunk oid 49 offset: 1073664 length: 1553408 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1073664 snap -1 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.531 INFO:tasks.rados.rados.0.smithi204.stdout:448: finishing set_chunk to oid 49 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1073664 length: 1553408 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:448: done (15 left) 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:449: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:449: set_chunk oid 50 offset: 1482752 length: 166400 target oid smithi20438004-50 low_tier offset: 1482752 snap -1 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:442: finishing set_chunk to oid 43 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1368064 length: 1139712 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:442: done (15 left) 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:450: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout:450: set_chunk oid 1 offset: 626688 length: 727552 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 626688 snap -1 2024-09-13T11:32:27.532 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:450: finishing set_chunk to oid 1 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 626688 length: 727552 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:450: done (15 left) 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:451: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:451: set_chunk oid 2 offset: 444928 length: 717824 target oid smithi20438004-2 low_tier offset: 444928 snap -1 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:451: finishing set_chunk to oid 2 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 444928 length: 717824 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:451: done (15 left) 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:452: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout:452: set_chunk oid 3 offset: 696320 length: 1024512 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 696320 snap -1 2024-09-13T11:32:27.533 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:449: finishing set_chunk to oid 50 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1482752 length: 166400 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:449: done (15 left) 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:453: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:453: set_chunk oid 4 offset: 388096 length: 1598976 target oid smithi20438004-4 low_tier offset: 388096 snap -1 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:412: finishing set_chunk to oid 13 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 182784 length: 1899520 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:412: done (15 left) 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:454: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:454: set_chunk oid 5 offset: 763904 length: 1005568 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 763904 snap -1 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.534 INFO:tasks.rados.rados.0.smithi204.stdout:423: finishing set_chunk to oid 24 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:423: done (15 left) 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:455: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:455: set_chunk oid 6 offset: 19968 length: 124416 target oid smithi20438004-6 low_tier offset: 19968 snap -1 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:444: finishing set_chunk to oid 45 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 503296 length: 1669120 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:444: done (15 left) 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:456: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout:456: set_chunk oid 7 offset: 725504 length: 12288 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 725504 snap -1 2024-09-13T11:32:27.535 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:454: finishing set_chunk to oid 5 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 763904 length: 1005568 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:453: finishing set_chunk to oid 4 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 388096 length: 1598976 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:453: done (15 left) 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:454: done (14 left) 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:457: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:457: set_chunk oid 8 offset: 1756672 length: 92160 target oid smithi20438004-8 low_tier offset: 1756672 snap -1 2024-09-13T11:32:27.536 INFO:tasks.rados.rados.0.smithi204.stdout:418: finishing set_chunk to oid 19 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:455: finishing set_chunk to oid 6 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 19968 length: 124416 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:418: done (14 left) 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:455: done (13 left) 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:458: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:458: set_chunk oid 9 offset: 312832 length: 707584 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 312832 snap -1 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:459: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:459: set_chunk oid 10 offset: 823808 length: 585216 target oid smithi20438004-10 low_tier offset: 823808 snap -1 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:459 retry set_chunk ! 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 15 2024-09-13T11:32:27.537 INFO:tasks.rados.rados.0.smithi204.stdout:452: finishing set_chunk to oid 3 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 696320 length: 1024512 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:452: done (14 left) 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 14 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:456: finishing set_chunk to oid 7 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 725504 length: 12288 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:456: done (13 left) 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 13 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:458: finishing set_chunk to oid 9 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 312832 length: 707584 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:459: finishing set_chunk to oid 10 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 823808 length: 585216 2024-09-13T11:32:27.538 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:458: done (12 left) 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:459: done (11 left) 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 11 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:428: finishing set_chunk to oid 29 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:428: done (10 left) 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:410: finishing set_chunk to oid 11 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:410: done (9 left) 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:427: finishing set_chunk to oid 28 2024-09-13T11:32:27.539 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:427: done (8 left) 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 8 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:425: finishing set_chunk to oid 26 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:425: done (7 left) 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 7 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:430: finishing set_chunk to oid 31 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:436: finishing set_chunk to oid 37 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 3 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:430: done (6 left) 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:436: done (5 left) 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.540 INFO:tasks.rados.rados.0.smithi204.stdout:415: finishing set_chunk to oid 16 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:415: done (4 left) 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:446: finishing set_chunk to oid 47 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:446: done (3 left) 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:447: finishing set_chunk to oid 48 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:447: done (2 left) 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:440: finishing set_chunk to oid 41 2024-09-13T11:32:27.541 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:440: done (1 left) 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:457: finishing set_chunk to oid 8 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:457: done (0 left) 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:460: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:460: set_chunk oid 11 offset: 4608 length: 1502720 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 4608 snap -1 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:461: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:461: set_chunk oid 12 offset: 1056256 length: 40960 target oid smithi20438004-12 low_tier offset: 1056256 snap -1 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:462: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:462: set_chunk oid 13 offset: 705536 length: 348672 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 705536 snap -1 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:463: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.542 INFO:tasks.rados.rados.0.smithi204.stdout:463: set_chunk oid 14 offset: 370688 length: 999424 target oid smithi20438004-14 low_tier offset: 370688 snap -1 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:464: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:464: set_chunk oid 15 offset: 806400 length: 213504 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 806400 snap -1 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:465: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:465: set_chunk oid 16 offset: 483328 length: 471040 target oid smithi20438004-16 low_tier offset: 483328 snap -1 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:466: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:466: set_chunk oid 17 offset: 1064960 length: 54272 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1064960 snap -1 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:467: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:467: set_chunk oid 18 offset: 1664000 length: 409600 target oid smithi20438004-18 low_tier offset: 1664000 snap -1 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:468: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:468: set_chunk oid 19 offset: 647680 length: 282624 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 647680 snap -1 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:469: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:469: set_chunk oid 20 offset: 472576 length: 1340928 target oid smithi20438004-20 low_tier offset: 472576 snap -1 2024-09-13T11:32:27.543 INFO:tasks.rados.rados.0.smithi204.stdout:470: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:470: set_chunk oid 21 offset: 62976 length: 1441280 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 62976 snap -1 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:471: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:471: set_chunk oid 22 offset: 1342976 length: 518656 target oid smithi20438004-22 low_tier offset: 1342976 snap -1 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:472: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:472: set_chunk oid 23 offset: 836608 length: 827904 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 836608 snap -1 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:473: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:473: set_chunk oid 24 offset: 1467392 length: 593408 target oid smithi20438004-24 low_tier offset: 1467392 snap -1 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:474: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:474: set_chunk oid 25 offset: 495104 length: 654336 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 495104 snap -1 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:475: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:475: set_chunk oid 26 offset: 1359872 length: 596480 target oid smithi20438004-26 low_tier offset: 1359872 snap -1 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.544 INFO:tasks.rados.rados.0.smithi204.stdout:460: finishing set_chunk to oid 11 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 4608 length: 1502720 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:460: done (15 left) 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:476: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:476: set_chunk oid 27 offset: 1127424 length: 828928 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1127424 snap -1 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:461: finishing set_chunk to oid 12 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1056256 length: 40960 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:461: done (15 left) 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:477: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:477: set_chunk oid 28 offset: 475648 length: 373248 target oid smithi20438004-28 low_tier offset: 475648 snap -1 2024-09-13T11:32:27.545 INFO:tasks.rados.rados.0.smithi204.stdout:463: finishing set_chunk to oid 14 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 370688 length: 999424 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:463: done (15 left) 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:478: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:478: set_chunk oid 29 offset: 972288 length: 802304 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 972288 snap -1 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:468: finishing set_chunk to oid 19 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 647680 length: 282624 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:472: finishing set_chunk to oid 23 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 836608 length: 827904 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:468: done (15 left) 2024-09-13T11:32:27.546 INFO:tasks.rados.rados.0.smithi204.stdout:472: done (14 left) 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:479: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:479: set_chunk oid 30 offset: 668160 length: 896000 target oid smithi20438004-30 low_tier offset: 668160 snap -1 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:480: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:480: set_chunk oid 31 offset: 95744 length: 1009152 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 95744 snap -1 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:473: finishing set_chunk to oid 24 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1467392 length: 593408 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:470: finishing set_chunk to oid 21 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 62976 length: 1441280 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:474: finishing set_chunk to oid 25 2024-09-13T11:32:27.547 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 495104 length: 654336 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:470: done (15 left) 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:473: done (14 left) 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:474: done (13 left) 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:481: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:469: finishing set_chunk to oid 20 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 472576 length: 1340928 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:475: finishing set_chunk to oid 26 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1359872 length: 596480 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:481: set_chunk oid 32 offset: 660992 length: 718848 target oid smithi20438004-32 low_tier offset: 660992 snap -1 2024-09-13T11:32:27.548 INFO:tasks.rados.rados.0.smithi204.stdout:469: done (13 left) 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:475: done (12 left) 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:482: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:482: set_chunk oid 33 offset: 671232 length: 598528 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 671232 snap -1 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:483: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:483: set_chunk oid 34 offset: 1958400 length: 97792 target oid smithi20438004-34 low_tier offset: 1958400 snap -1 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:484: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:477: finishing set_chunk to oid 28 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 475648 length: 373248 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:476: finishing set_chunk to oid 27 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1127424 length: 828928 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.549 INFO:tasks.rados.rados.0.smithi204.stdout:484: set_chunk oid 35 offset: 102400 length: 1402880 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 102400 snap -1 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:476: done (14 left) 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:477: done (13 left) 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:485: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:485: set_chunk oid 36 offset: 223744 length: 654848 target oid smithi20438004-36 low_tier offset: 223744 snap -1 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:486: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:486: set_chunk oid 37 offset: 1998848 length: 492032 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1998848 snap -1 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:487: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:487: set_chunk oid 38 offset: 72704 length: 726528 target oid smithi20438004-38 low_tier offset: 72704 snap -1 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:478: finishing set_chunk to oid 29 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 972288 length: 802304 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:464: finishing set_chunk to oid 15 2024-09-13T11:32:27.550 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 806400 length: 213504 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:467: finishing set_chunk to oid 18 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1664000 length: 409600 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:464: done (15 left) 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:467: done (14 left) 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:478: done (13 left) 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:488: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:488: set_chunk oid 39 offset: 2330624 length: 262144 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2330624 snap -1 2024-09-13T11:32:27.551 INFO:tasks.rados.rados.0.smithi204.stdout:489: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:489: set_chunk oid 40 offset: 87552 length: 488960 target oid smithi20438004-40 low_tier offset: 87552 snap -1 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:490: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:490: set_chunk oid 41 offset: 875008 length: 1688576 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 875008 snap -1 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:465: finishing set_chunk to oid 16 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 483328 length: 471040 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:466: finishing set_chunk to oid 17 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1064960 length: 54272 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:481: finishing set_chunk to oid 32 2024-09-13T11:32:27.552 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 660992 length: 718848 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:462: finishing set_chunk to oid 13 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 705536 length: 348672 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:479: finishing set_chunk to oid 30 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 668160 length: 896000 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:485: finishing set_chunk to oid 36 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 223744 length: 654848 2024-09-13T11:32:27.553 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:482: finishing set_chunk to oid 33 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 671232 length: 598528 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:462: done (15 left) 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:465: done (14 left) 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:466: done (13 left) 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:479: done (12 left) 2024-09-13T11:32:27.554 INFO:tasks.rados.rados.0.smithi204.stdout:481: done (11 left) 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:482: done (10 left) 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:485: done (9 left) 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:491: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:491: set_chunk oid 42 offset: 630784 length: 628224 target oid smithi20438004-42 low_tier offset: 630784 snap -1 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:492: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:492: set_chunk oid 43 offset: 692736 length: 1932800 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 692736 snap -1 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:493: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:493: set_chunk oid 44 offset: 154624 length: 199680 target oid smithi20438004-44 low_tier offset: 154624 snap -1 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:494: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:494: set_chunk oid 45 offset: 491520 length: 776704 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 491520 snap -1 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:495: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.555 INFO:tasks.rados.rados.0.smithi204.stdout:495: set_chunk oid 46 offset: 851968 length: 1391104 target oid smithi20438004-46 low_tier offset: 851968 snap -1 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:496: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:496: set_chunk oid 47 offset: 189952 length: 1007104 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 189952 snap -1 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:497: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:497: set_chunk oid 48 offset: 535040 length: 245760 target oid smithi20438004-48 low_tier offset: 535040 snap -1 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:487: finishing set_chunk to oid 38 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 72704 length: 726528 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:471: finishing set_chunk to oid 22 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1342976 length: 518656 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:471: done (15 left) 2024-09-13T11:32:27.556 INFO:tasks.rados.rados.0.smithi204.stdout:487: done (14 left) 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:498: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:498: set_chunk oid 49 offset: 1172480 length: 193536 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1172480 snap -1 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:499: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:499: set_chunk oid 50 offset: 1974784 length: 813568 target oid smithi20438004-50 low_tier offset: 1974784 snap -1 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:480: finishing set_chunk to oid 31 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 95744 length: 1009152 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:480: done (15 left) 2024-09-13T11:32:27.557 INFO:tasks.rados.rados.0.smithi204.stdout:500: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:500: set_chunk oid 1 offset: 375808 length: 213504 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 375808 snap -1 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:491: finishing set_chunk to oid 42 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 630784 length: 628224 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:491: done (15 left) 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:501: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:501: set_chunk oid 2 offset: 1368064 length: 489984 target oid smithi20438004-2 low_tier offset: 1368064 snap -1 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:495: finishing set_chunk to oid 46 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 851968 length: 1391104 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.558 INFO:tasks.rados.rados.0.smithi204.stdout:495: done (15 left) 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:502: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:502: set_chunk oid 3 offset: 933376 length: 806912 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 933376 snap -1 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:486: finishing set_chunk to oid 37 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1998848 length: 492032 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:489: finishing set_chunk to oid 40 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 87552 length: 488960 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:486: done (15 left) 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:489: done (14 left) 2024-09-13T11:32:27.559 INFO:tasks.rados.rados.0.smithi204.stdout:503: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:503: set_chunk oid 4 offset: 439296 length: 884736 target oid smithi20438004-4 low_tier offset: 439296 snap -1 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:504: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:504: set_chunk oid 5 offset: 475648 length: 1313792 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 475648 snap -1 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:494: finishing set_chunk to oid 45 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 491520 length: 776704 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:484: finishing set_chunk to oid 35 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 102400 length: 1402880 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:497: finishing set_chunk to oid 48 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 535040 length: 245760 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.560 INFO:tasks.rados.rados.0.smithi204.stdout:501: finishing set_chunk to oid 2 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1368064 length: 489984 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:496: finishing set_chunk to oid 47 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 189952 length: 1007104 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:490: finishing set_chunk to oid 41 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 875008 length: 1688576 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:484: done (15 left) 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:490: done (14 left) 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:494: done (13 left) 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:496: done (12 left) 2024-09-13T11:32:27.561 INFO:tasks.rados.rados.0.smithi204.stdout:497: done (11 left) 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:501: done (10 left) 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:505: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:505: set_chunk oid 6 offset: 1446400 length: 144384 target oid smithi20438004-6 low_tier offset: 1446400 snap -1 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:506: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:506: set_chunk oid 7 offset: 737792 length: 225792 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 737792 snap -1 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:502: finishing set_chunk to oid 3 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 933376 length: 806912 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:492: finishing set_chunk to oid 43 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 692736 length: 1932800 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:500: finishing set_chunk to oid 1 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 375808 length: 213504 2024-09-13T11:32:27.562 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:492: done (11 left) 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:500: done (10 left) 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:502: done (9 left) 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:507: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:507: set_chunk oid 8 offset: 1189376 length: 858112 target oid smithi20438004-8 low_tier offset: 1189376 snap -1 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:503: finishing set_chunk to oid 4 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 439296 length: 884736 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:503: done (9 left) 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:508: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:508: set_chunk oid 9 offset: 446976 length: 525312 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 446976 snap -1 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:509: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:509: set_chunk oid 10 offset: 1092096 length: 225792 target oid smithi20438004-10 low_tier offset: 1092096 snap -1 2024-09-13T11:32:27.563 INFO:tasks.rados.rados.0.smithi204.stdout:510: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:510: set_chunk oid 11 offset: 316416 length: 1648128 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 316416 snap -1 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:511: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:511: set_chunk oid 12 offset: 89088 length: 1327616 target oid smithi20438004-12 low_tier offset: 89088 snap -1 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:512: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:512: set_chunk oid 13 offset: 675840 length: 1300992 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 675840 snap -1 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:513: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:513: set_chunk oid 14 offset: 825856 length: 195072 target oid smithi20438004-14 low_tier offset: 825856 snap -1 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:514: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:514: set_chunk oid 15 offset: 378368 length: 1416704 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 378368 snap -1 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:505: finishing set_chunk to oid 6 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1446400 length: 144384 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:504: finishing set_chunk to oid 5 2024-09-13T11:32:27.564 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 475648 length: 1313792 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:504: done (15 left) 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:505: done (14 left) 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:515: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:515: set_chunk oid 16 offset: 612352 length: 606208 target oid smithi20438004-16 low_tier offset: 612352 snap -1 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:516: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:516: set_chunk oid 17 offset: 326144 length: 1860608 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 326144 snap -1 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:506: finishing set_chunk to oid 7 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 737792 length: 225792 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:506: done (15 left) 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:517: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.565 INFO:tasks.rados.rados.0.smithi204.stdout:517: set_chunk oid 18 offset: 1462784 length: 213504 target oid smithi20438004-18 low_tier offset: 1462784 snap -1 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:507: finishing set_chunk to oid 8 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1189376 length: 858112 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:511: finishing set_chunk to oid 12 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 89088 length: 1327616 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:507: done (15 left) 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:511: done (14 left) 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:518: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:499: finishing set_chunk to oid 50 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1974784 length: 813568 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.566 INFO:tasks.rados.rados.0.smithi204.stdout:518: set_chunk oid 19 offset: 684032 length: 913920 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 684032 snap -1 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:499: done (14 left) 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:519: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:519: set_chunk oid 20 offset: 770560 length: 1402368 target oid smithi20438004-20 low_tier offset: 770560 snap -1 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:520: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:520: set_chunk oid 21 offset: 196608 length: 2004992 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 196608 snap -1 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:508: finishing set_chunk to oid 9 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 446976 length: 525312 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:509: finishing set_chunk to oid 10 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 1092096 length: 225792 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:512: finishing set_chunk to oid 13 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 675840 length: 1300992 2024-09-13T11:32:27.567 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:508: done (15 left) 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:509: done (14 left) 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:512: done (13 left) 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:521: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:521: set_chunk oid 22 offset: 854528 length: 803328 target oid smithi20438004-22 low_tier offset: 854528 snap -1 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:522: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:522: set_chunk oid 23 offset: 1087488 length: 992768 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1087488 snap -1 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:523: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:523: set_chunk oid 24 offset: 850432 length: 1116672 target oid smithi20438004-24 low_tier offset: 850432 snap -1 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.568 INFO:tasks.rados.rados.0.smithi204.stdout:510: finishing set_chunk to oid 11 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 316416 length: 1648128 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:510: done (15 left) 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:524: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:524: set_chunk oid 25 offset: 610304 length: 1391616 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 610304 snap -1 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:518: finishing set_chunk to oid 19 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 684032 length: 913920 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.569 INFO:tasks.rados.rados.0.smithi204.stdout:518: done (15 left) 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:525: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:525: set_chunk oid 26 offset: 1549312 length: 54272 target oid smithi20438004-26 low_tier offset: 1549312 snap -1 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:514: finishing set_chunk to oid 15 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 378368 length: 1416704 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:516: finishing set_chunk to oid 17 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 326144 length: 1860608 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:519: finishing set_chunk to oid 20 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 770560 length: 1402368 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.570 INFO:tasks.rados.rados.0.smithi204.stdout:514: done (15 left) 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:516: done (14 left) 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:519: done (13 left) 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:526: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:513: finishing set_chunk to oid 14 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 825856 length: 195072 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:526: set_chunk oid 27 offset: 344064 length: 1070080 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 344064 snap -1 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:513: done (13 left) 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:527: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:527: set_chunk oid 28 offset: 898048 length: 1330688 target oid smithi20438004-28 low_tier offset: 898048 snap -1 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:528: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:522: finishing set_chunk to oid 23 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1087488 length: 992768 2024-09-13T11:32:27.571 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:520: finishing set_chunk to oid 21 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 196608 length: 2004992 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:524: finishing set_chunk to oid 25 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 610304 length: 1391616 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:528: set_chunk oid 29 offset: 850432 length: 967680 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 850432 snap -1 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:520: done (14 left) 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:522: done (13 left) 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:524: done (12 left) 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:529: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:529: set_chunk oid 30 offset: 817152 length: 467456 target oid smithi20438004-30 low_tier offset: 817152 snap -1 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:521: finishing set_chunk to oid 22 2024-09-13T11:32:27.572 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 854528 length: 803328 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:523: finishing set_chunk to oid 24 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 850432 length: 1116672 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:521: done (12 left) 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:523: done (11 left) 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:530: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:515: finishing set_chunk to oid 16 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 612352 length: 606208 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:530: set_chunk oid 31 offset: 797696 length: 1120256 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 797696 snap -1 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:515: done (11 left) 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:531: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.573 INFO:tasks.rados.rados.0.smithi204.stdout:531: set_chunk oid 32 offset: 72192 length: 396288 target oid smithi20438004-32 low_tier offset: 72192 snap -1 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:532: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:532: set_chunk oid 33 offset: 684544 length: 1547264 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 684544 snap -1 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:532 retry set_chunk ! 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 13 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:525: finishing set_chunk to oid 26 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1549312 length: 54272 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:525: done (12 left) 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 12 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:527: finishing set_chunk to oid 28 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 898048 length: 1330688 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:527: done (11 left) 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 11 2024-09-13T11:32:27.574 INFO:tasks.rados.rados.0.smithi204.stdout:529: finishing set_chunk to oid 30 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 817152 length: 467456 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:529: done (10 left) 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:528: finishing set_chunk to oid 29 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 850432 length: 967680 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:528: done (9 left) 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:531: finishing set_chunk to oid 32 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 72192 length: 396288 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:530: finishing set_chunk to oid 31 2024-09-13T11:32:27.575 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 797696 length: 1120256 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:526: finishing set_chunk to oid 27 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 344064 length: 1070080 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:526: done (8 left) 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:530: done (7 left) 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:531: done (6 left) 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:532: finishing set_chunk to oid 33 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 684544 length: 1547264 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout:532: done (5 left) 2024-09-13T11:32:27.576 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:488: finishing set_chunk to oid 39 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:488: done (4 left) 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:498: finishing set_chunk to oid 49 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 6 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:498: done (3 left) 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:483: finishing set_chunk to oid 34 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:483: done (2 left) 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:493: finishing set_chunk to oid 44 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.577 INFO:tasks.rados.rados.0.smithi204.stdout:493: done (1 left) 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:517: finishing set_chunk to oid 18 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:517: done (0 left) 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:533: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:533: set_chunk oid 34 offset: 263168 length: 535552 target oid smithi20438004-34 low_tier offset: 263168 snap -1 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:534: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:534: set_chunk oid 35 offset: 763904 length: 318464 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 763904 snap -1 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:535: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:535: set_chunk oid 36 offset: 1713664 length: 523264 target oid smithi20438004-36 low_tier offset: 1713664 snap -1 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:536: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.578 INFO:tasks.rados.rados.0.smithi204.stdout:536: set_chunk oid 37 offset: 207360 length: 2124800 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 207360 snap -1 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:537: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:537: set_chunk oid 38 offset: 1072128 length: 299008 target oid smithi20438004-38 low_tier offset: 1072128 snap -1 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:538: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:538: set_chunk oid 39 offset: 193024 length: 1589760 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 193024 snap -1 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:539: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:539: set_chunk oid 40 offset: 617984 length: 1729536 target oid smithi20438004-40 low_tier offset: 617984 snap -1 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:540: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:540: set_chunk oid 41 offset: 512512 length: 1443840 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 512512 snap -1 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:541: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:541: set_chunk oid 42 offset: 573440 length: 905216 target oid smithi20438004-42 low_tier offset: 573440 snap -1 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:542: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:542: set_chunk oid 43 offset: 732672 length: 1945600 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 732672 snap -1 2024-09-13T11:32:27.579 INFO:tasks.rados.rados.0.smithi204.stdout:543: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:543: set_chunk oid 44 offset: 211456 length: 282624 target oid smithi20438004-44 low_tier offset: 211456 snap -1 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:544: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:544: set_chunk oid 45 offset: 261120 length: 849408 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 261120 snap -1 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:545: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:545: set_chunk oid 46 offset: 422912 length: 1384960 target oid smithi20438004-46 low_tier offset: 422912 snap -1 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:546: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:546: set_chunk oid 47 offset: 1357824 length: 40960 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1357824 snap -1 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:547: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:547: set_chunk oid 48 offset: 1610752 length: 619008 target oid smithi20438004-48 low_tier offset: 1610752 snap -1 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:548: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout:548: set_chunk oid 49 offset: 543744 length: 1011200 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 543744 snap -1 2024-09-13T11:32:27.580 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:538: finishing set_chunk to oid 39 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 193024 length: 1589760 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:538: done (15 left) 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:549: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:549: set_chunk oid 50 offset: 652288 length: 796160 target oid smithi20438004-50 low_tier offset: 652288 snap -1 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:544: finishing set_chunk to oid 45 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 261120 length: 849408 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:541: finishing set_chunk to oid 42 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 573440 length: 905216 2024-09-13T11:32:27.581 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:541: done (15 left) 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:544: done (14 left) 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:550: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:550: set_chunk oid 1 offset: 22528 length: 1902592 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 22528 snap -1 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:551: set_chunk oid 2 target oid 2 low_tier 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:551: set_chunk oid 2 offset: 691712 length: 315392 target oid smithi20438004-2 low_tier offset: 691712 snap -1 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:543: finishing set_chunk to oid 44 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 211456 length: 282624 2024-09-13T11:32:27.582 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:545: finishing set_chunk to oid 46 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 422912 length: 1384960 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:543: done (15 left) 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:545: done (14 left) 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:552: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:552: set_chunk oid 3 offset: 263680 length: 350208 target oid smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 263680 snap -1 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:553: set_chunk oid 4 target oid 4 low_tier 2024-09-13T11:32:27.583 INFO:tasks.rados.rados.0.smithi204.stdout:553: set_chunk oid 4 offset: 436224 length: 913920 target oid smithi20438004-4 low_tier offset: 436224 snap -1 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:533: finishing set_chunk to oid 34 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 263168 length: 535552 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:539: finishing set_chunk to oid 40 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 617984 length: 1729536 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:536: finishing set_chunk to oid 37 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 207360 length: 2124800 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:534: finishing set_chunk to oid 35 2024-09-13T11:32:27.584 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 763904 length: 318464 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:548: finishing set_chunk to oid 49 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 543744 length: 1011200 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:547: finishing set_chunk to oid 48 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1610752 length: 619008 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:533: done (15 left) 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:534: done (14 left) 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:536: done (13 left) 2024-09-13T11:32:27.585 INFO:tasks.rados.rados.0.smithi204.stdout:539: done (12 left) 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:547: done (11 left) 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:548: done (10 left) 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:554: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:554: set_chunk oid 5 offset: 297472 length: 866816 target oid smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 297472 snap -1 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:555: set_chunk oid 6 target oid 6 low_tier 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:555: set_chunk oid 6 offset: 570368 length: 569344 target oid smithi20438004-6 low_tier offset: 570368 snap -1 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:556: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:556: set_chunk oid 7 offset: 1246720 length: 340992 target oid smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1246720 snap -1 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:557: set_chunk oid 8 target oid 8 low_tier 2024-09-13T11:32:27.586 INFO:tasks.rados.rados.0.smithi204.stdout:557: set_chunk oid 8 offset: 673280 length: 1057792 target oid smithi20438004-8 low_tier offset: 673280 snap -1 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:558: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:558: set_chunk oid 9 offset: 320000 length: 1149952 target oid smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 320000 snap -1 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:559: set_chunk oid 10 target oid 10 low_tier 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:559: set_chunk oid 10 offset: 353280 length: 797184 target oid smithi20438004-10 low_tier offset: 353280 snap -1 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:546: finishing set_chunk to oid 47 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1357824 length: 40960 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:540: finishing set_chunk to oid 41 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 512512 length: 1443840 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:551: finishing set_chunk to oid 2 2024-09-13T11:32:27.587 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 691712 length: 315392 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:549: finishing set_chunk to oid 50 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 652288 length: 796160 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:550: finishing set_chunk to oid 1 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 22528 length: 1902592 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:540: done (15 left) 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:546: done (14 left) 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:549: done (13 left) 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:550: done (12 left) 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:551: done (11 left) 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:560: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.588 INFO:tasks.rados.rados.0.smithi204.stdout:560: set_chunk oid 11 offset: 543232 length: 1233920 target oid smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 543232 snap -1 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:561: set_chunk oid 12 target oid 12 low_tier 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:561: set_chunk oid 12 offset: 868864 length: 847360 target oid smithi20438004-12 low_tier offset: 868864 snap -1 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:562: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:562: set_chunk oid 13 offset: 1114624 length: 268288 target oid smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1114624 snap -1 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:563: set_chunk oid 14 target oid 14 low_tier 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:563: set_chunk oid 14 offset: 11264 length: 157184 target oid smithi20438004-14 low_tier offset: 11264 snap -1 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:564: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:564: set_chunk oid 15 offset: 177152 length: 1661952 target oid smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 177152 snap -1 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:542: finishing set_chunk to oid 43 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 732672 length: 1945600 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:553: finishing set_chunk to oid 4 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 436224 length: 913920 2024-09-13T11:32:27.589 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:552: finishing set_chunk to oid 3 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 263680 length: 350208 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:542: done (15 left) 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:552: done (14 left) 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:553: done (13 left) 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:565: set_chunk oid 16 target oid 16 low_tier 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:565: set_chunk oid 16 offset: 295936 length: 1084928 target oid smithi20438004-16 low_tier offset: 295936 snap -1 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:566: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:566: set_chunk oid 17 offset: 1707008 length: 356864 target oid smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1707008 snap -1 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:567: set_chunk oid 18 target oid 18 low_tier 2024-09-13T11:32:27.590 INFO:tasks.rados.rados.0.smithi204.stdout:567: set_chunk oid 18 offset: 1136640 length: 554496 target oid smithi20438004-18 low_tier offset: 1136640 snap -1 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:557: finishing set_chunk to oid 8 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 673280 length: 1057792 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:554: finishing set_chunk to oid 5 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 297472 length: 866816 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:556: finishing set_chunk to oid 7 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1246720 length: 340992 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:554: done (15 left) 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:556: done (14 left) 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:557: done (13 left) 2024-09-13T11:32:27.591 INFO:tasks.rados.rados.0.smithi204.stdout:568: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:568: set_chunk oid 19 offset: 947200 length: 241664 target oid smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 947200 snap -1 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:569: set_chunk oid 20 target oid 20 low_tier 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:569: set_chunk oid 20 offset: 281088 length: 369664 target oid smithi20438004-20 low_tier offset: 281088 snap -1 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:570: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:570: set_chunk oid 21 offset: 387584 length: 665088 target oid smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 387584 snap -1 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:559: finishing set_chunk to oid 10 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 353280 length: 797184 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:558: finishing set_chunk to oid 9 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 320000 length: 1149952 2024-09-13T11:32:27.592 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:562: finishing set_chunk to oid 13 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1114624 length: 268288 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:560: finishing set_chunk to oid 11 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 543232 length: 1233920 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:555: finishing set_chunk to oid 6 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 570368 length: 569344 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:561: finishing set_chunk to oid 12 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 868864 length: 847360 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:555: done (15 left) 2024-09-13T11:32:27.593 INFO:tasks.rados.rados.0.smithi204.stdout:558: done (14 left) 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:559: done (13 left) 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:560: done (12 left) 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:561: done (11 left) 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:562: done (10 left) 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:571: set_chunk oid 22 target oid 22 low_tier 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:571: set_chunk oid 22 offset: 180736 length: 261632 target oid smithi20438004-22 low_tier offset: 180736 snap -1 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:572: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:572: set_chunk oid 23 offset: 1108992 length: 374784 target oid smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1108992 snap -1 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:573: set_chunk oid 24 target oid 24 low_tier 2024-09-13T11:32:27.594 INFO:tasks.rados.rados.0.smithi204.stdout:573: set_chunk oid 24 offset: 448512 length: 1740800 target oid smithi20438004-24 low_tier offset: 448512 snap -1 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:574: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:574: set_chunk oid 25 offset: 1250816 length: 190976 target oid smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1250816 snap -1 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:575: set_chunk oid 26 target oid 26 low_tier 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:575: set_chunk oid 26 offset: 1131520 length: 622592 target oid smithi20438004-26 low_tier offset: 1131520 snap -1 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:576: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:576: set_chunk oid 27 offset: 363520 length: 1884672 target oid smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 363520 snap -1 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:564: finishing set_chunk to oid 15 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 177152 length: 1661952 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:565: finishing set_chunk to oid 16 2024-09-13T11:32:27.595 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 295936 length: 1084928 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:563: finishing set_chunk to oid 14 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 11264 length: 157184 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:568: finishing set_chunk to oid 19 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 947200 length: 241664 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:563: done (15 left) 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:564: done (14 left) 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:565: done (13 left) 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:568: done (12 left) 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:577: set_chunk oid 28 target oid 28 low_tier 2024-09-13T11:32:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:577: set_chunk oid 28 offset: 196096 length: 1288192 target oid smithi20438004-28 low_tier offset: 196096 snap -1 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:578: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:578: set_chunk oid 29 offset: 557568 length: 294400 target oid smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 557568 snap -1 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:579: set_chunk oid 30 target oid 30 low_tier 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:579: set_chunk oid 30 offset: 653824 length: 1509376 target oid smithi20438004-30 low_tier offset: 653824 snap -1 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:580: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:580: set_chunk oid 31 offset: 1783296 length: 585728 target oid smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1783296 snap -1 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:567: finishing set_chunk to oid 18 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1136640 length: 554496 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:572: finishing set_chunk to oid 23 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1108992 length: 374784 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-13T11:32:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:566: finishing set_chunk to oid 17 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1707008 length: 356864 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:571: finishing set_chunk to oid 22 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 180736 length: 261632 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:566: done (15 left) 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:567: done (14 left) 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:571: done (13 left) 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:572: done (12 left) 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:581: set_chunk oid 32 target oid 32 low_tier 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:581: set_chunk oid 32 offset: 219136 length: 1465344 target oid smithi20438004-32 low_tier offset: 219136 snap -1 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:582: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.598 INFO:tasks.rados.rados.0.smithi204.stdout:582: set_chunk oid 33 offset: 73216 length: 197120 target oid smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 73216 snap -1 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:583: set_chunk oid 34 target oid 34 low_tier 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:583: set_chunk oid 34 offset: 1281536 length: 489472 target oid smithi20438004-34 low_tier offset: 1281536 snap -1 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:584: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:584: set_chunk oid 35 offset: 1009152 length: 878080 target oid smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1009152 snap -1 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:570: finishing set_chunk to oid 21 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 387584 length: 665088 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:573: finishing set_chunk to oid 24 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 448512 length: 1740800 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:574: finishing set_chunk to oid 25 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1250816 length: 190976 2024-09-13T11:32:27.599 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:575: finishing set_chunk to oid 26 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1131520 length: 622592 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:578: finishing set_chunk to oid 29 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 557568 length: 294400 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:579: finishing set_chunk to oid 30 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 653824 length: 1509376 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:570: done (15 left) 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:573: done (14 left) 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:574: done (13 left) 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:575: done (12 left) 2024-09-13T11:32:27.600 INFO:tasks.rados.rados.0.smithi204.stdout:578: done (11 left) 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:579: done (10 left) 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:584 retry set_chunk ! 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:580: finishing set_chunk to oid 31 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1783296 length: 585728 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:580: done (9 left) 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 9 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:576: finishing set_chunk to oid 27 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 363520 length: 1884672 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.601 INFO:tasks.rados.rados.0.smithi204.stdout:583: finishing set_chunk to oid 34 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1281536 length: 489472 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:576: done (8 left) 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:583: done (7 left) 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 7 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:581: finishing set_chunk to oid 32 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 219136 length: 1465344 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:581: done (6 left) 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:582: finishing set_chunk to oid 33 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 73216 length: 197120 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-13T11:32:27.602 INFO:tasks.rados.rados.0.smithi204.stdout:582: done (5 left) 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:577: finishing set_chunk to oid 28 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 196096 length: 1288192 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:577: done (4 left) 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:535: finishing set_chunk to oid 36 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:535: done (3 left) 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:537: finishing set_chunk to oid 38 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:537: done (2 left) 2024-09-13T11:32:27.603 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:584: finishing set_chunk to oid 35 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:584: done (1 left) 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:569: finishing set_chunk to oid 20 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:569: done (0 left) 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:585: set_chunk oid 36 target oid 36 low_tier 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:585: set_chunk oid 36 offset: 28160 length: 41472 target oid smithi20438004-36 low_tier offset: 28160 snap -1 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:586: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:586: set_chunk oid 37 offset: 823808 length: 1221632 target oid smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 823808 snap -1 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:587: set_chunk oid 38 target oid 38 low_tier 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:587: set_chunk oid 38 offset: 1839104 length: 548864 target oid smithi20438004-38 low_tier offset: 1839104 snap -1 2024-09-13T11:32:27.604 INFO:tasks.rados.rados.0.smithi204.stdout:588: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:588: set_chunk oid 39 offset: 322048 length: 1419264 target oid smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 322048 snap -1 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:589: set_chunk oid 40 target oid 40 low_tier 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:589: set_chunk oid 40 offset: 19968 length: 1647104 target oid smithi20438004-40 low_tier offset: 19968 snap -1 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:590: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:590: set_chunk oid 41 offset: 2090496 length: 224256 target oid smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2090496 snap -1 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:591: set_chunk oid 42 target oid 42 low_tier 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:591: set_chunk oid 42 offset: 676864 length: 360960 target oid smithi20438004-42 low_tier offset: 676864 snap -1 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:592: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:592: set_chunk oid 43 offset: 1442816 length: 189440 target oid smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1442816 snap -1 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:593: set_chunk oid 44 target oid 44 low_tier 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:593: set_chunk oid 44 offset: 944640 length: 1681920 target oid smithi20438004-44 low_tier offset: 944640 snap -1 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:594: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.605 INFO:tasks.rados.rados.0.smithi204.stdout:594: set_chunk oid 45 offset: 1250304 length: 1052160 target oid smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1250304 snap -1 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:595: set_chunk oid 46 target oid 46 low_tier 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:595: set_chunk oid 46 offset: 567808 length: 969216 target oid smithi20438004-46 low_tier offset: 567808 snap -1 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:596: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:596: set_chunk oid 47 offset: 1514496 length: 1031680 target oid smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1514496 snap -1 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:597: set_chunk oid 48 target oid 48 low_tier 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:597: set_chunk oid 48 offset: 1252352 length: 135168 target oid smithi20438004-48 low_tier offset: 1252352 snap -1 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:598: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:598: set_chunk oid 49 offset: 920576 length: 1780224 target oid smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 920576 snap -1 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:599: set_chunk oid 50 target oid 50 low_tier 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:599: set_chunk oid 50 offset: 876544 length: 1682944 target oid smithi20438004-50 low_tier offset: 876544 snap -1 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:600: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:585: finishing set_chunk to oid 36 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 28160 length: 41472 2024-09-13T11:32:27.606 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:600: set_chunk oid 1 offset: 483328 length: 610816 target oid smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 483328 snap -1 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:585: done (15 left) 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:601 oid_not_in_use 35 oid_redirect_not_in_use 0 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:600 wait for manifest initialization 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:588: finishing set_chunk to oid 39 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 322048 length: 1419264 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:588: done (14 left) 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 14 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:591: finishing set_chunk to oid 42 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 676864 length: 360960 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout:591: done (13 left) 2024-09-13T11:32:27.607 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 13 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:587: finishing set_chunk to oid 38 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1839104 length: 548864 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:593: finishing set_chunk to oid 44 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 944640 length: 1681920 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:587: done (12 left) 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:593: done (11 left) 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 11 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:595: finishing set_chunk to oid 46 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 567808 length: 969216 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:27.608 INFO:tasks.rados.rados.0.smithi204.stdout:595: done (10 left) 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:598: finishing set_chunk to oid 49 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 920576 length: 1780224 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:586: finishing set_chunk to oid 37 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 823808 length: 1221632 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:590: finishing set_chunk to oid 41 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2090496 length: 224256 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:589: finishing set_chunk to oid 40 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 19968 length: 1647104 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:27.609 INFO:tasks.rados.rados.0.smithi204.stdout:586: done (9 left) 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:589: done (8 left) 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:590: done (7 left) 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:598: done (6 left) 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 6 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:592: finishing set_chunk to oid 43 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1442816 length: 189440 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:592: done (5 left) 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 5 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:596: finishing set_chunk to oid 47 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1514496 length: 1031680 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.610 INFO:tasks.rados.rados.0.smithi204.stdout:596: done (4 left) 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:599: finishing set_chunk to oid 50 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 876544 length: 1682944 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:599: done (3 left) 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:597: finishing set_chunk to oid 48 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1252352 length: 135168 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:597: done (2 left) 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:594: finishing set_chunk to oid 45 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1250304 length: 1052160 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-13T11:32:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:594: done (1 left) 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout:600: finishing set_chunk to oid 1 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 483328 length: 610816 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout:600: done (0 left) 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout:601 oid_not_in_use 50 oid_redirect_not_in_use 0 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 51 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 52 low_tier 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 53 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 54 low_tier 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 55 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 56 low_tier 2024-09-13T11:32:27.612 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 57 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 58 low_tier 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 59 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 60 low_tier 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 61 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 62 low_tier 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 63 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 64 low_tier 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 65 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 66 low_tier 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 68 low_tier 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 69 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.613 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 70 low_tier 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 71 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 72 low_tier 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 73 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 74 low_tier 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 75 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 76 low_tier 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 77 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 78 low_tier 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 79 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 80 low_tier 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 81 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 82 low_tier 2024-09-13T11:32:27.614 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 83 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 84 low_tier 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 85 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 86 low_tier 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 87 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 88 low_tier 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 89 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 90 low_tier 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 91 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 92 low_tier 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 93 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 94 low_tier 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 95 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 96 low_tier 2024-09-13T11:32:27.615 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 97 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 98 low_tier 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 99 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout: redirect_not_in_use: 100 low_tier 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:601: read oid 43 snap -1 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:601: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:602: read oid 26 snap -1 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:602: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:603: read oid 49 snap -1 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:603: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:604: set_chunk oid 25 target oid 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:604: set_chunk oid 25 offset: 995840 length: 239616 target oid 155ec6f051597d74dda2d4e7532cef68cad136471e633363914261da370bd34d offset: 0 snap -1 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:605: write oid 50 current snap is 0 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:605: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:605: writing smithi20438004-50 from 440350 to 900918 tid 1 2024-09-13T11:32:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:605: writing smithi20438004-50 from 1309629 to 1852295 tid 2 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:605: writing smithi20438004-50 from 2323973 to 2840350 tid 3 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:606: read oid 47 snap -1 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:606: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:607: set_chunk oid 29 target oid 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:607: set_chunk oid 29 offset: 1014784 length: 1380864 target oid 4ec2dd001b7153a0bb00c76a8320dec150c3eb87238ccdd9135be5e948aee2d9 offset: 0 snap -1 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:608: set_chunk oid 17 target oid 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:608: set_chunk oid 17 offset: 463872 length: 642048 target oid ad2ee76ff1a2885698d2b6c37a67b612678c43b97b5b65c8c121c154cdf4f074 offset: 0 snap -1 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:609: set_chunk oid 27 target oid 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:609: set_chunk oid 27 offset: 1058304 length: 285696 target oid 86a7696fc457e5e76f49cd06bb2ec48205039b5bf599e041c73e1b4cb7a9729c offset: 0 snap -1 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:610: write oid 38 current snap is 0 2024-09-13T11:32:27.617 INFO:tasks.rados.rados.0.smithi204.stdout:610: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-13T11:32:27.620 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:32:27.621 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:27.621 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:27.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:27.621 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:57 4.4 e38/31 4:28ee04de:::smithi20438004-15:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.621288+0000", 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.80804867000000002, 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004995083, 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.621288+0000", 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.621288+0000", 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.621288+0000", 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6300000000000001e-07 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.622245+0000", 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00095609199999999997 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.622256+0000", 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1105e-05 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.622264+0000", 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8249999999999994e-06 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.622320+0000", 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5498999999999998e-05 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.622338+0000", 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8009999999999999e-05 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.626241+0000", 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039031209999999998 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.626283+0000", 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2168e-05 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:67 4.4 e38/31 4:2451796c:::smithi20438004-17:head v 38'6, mlcod=38'3)", 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.649736+0000", 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.77960048599999998, 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031452479999999998, 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.626 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649736+0000", 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649736+0000", 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649737+0000", 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6699999999999999e-07 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649743+0000", 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2890000000000003e-06 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649749+0000", 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1999999999999999e-06 2024-09-13T11:32:27.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649753+0000", 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6569999999999999e-06 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649866+0000", 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112935 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.649880+0000", 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3769e-05 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.652855+0000", 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029756779999999998 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.652882+0000", 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6253000000000001e-05 2024-09-13T11:32:27.630 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:85 4.4 e38/31 4:2c957d44:::smithi20438004-22:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.708505+0000", 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.72083169300000005, 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020525169999999998, 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708505+0000", 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708505+0000", 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708506+0000", 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1659999999999999e-06 2024-09-13T11:32:27.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708772+0000", 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026519 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708784+0000", 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.27e-05 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708795+0000", 2024-09-13T11:32:27.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1226000000000001e-05 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708847+0000", 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1051999999999998e-05 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.708864+0000", 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7065e-05 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.710522+0000", 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001658893 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.710558+0000", 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5225000000000002e-05 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:157 4.4 e38/31 4:2b094039:::smithi20438004-40:head v 38'10, mlcod=38'8)", 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.936756+0000", 2024-09-13T11:32:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.49258067500000002, 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018874969999999999, 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.936756+0000", 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.936756+0000", 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.936757+0000", 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.156e-06 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.937452+0000", 2024-09-13T11:32:27.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069470199999999995 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.937467+0000", 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4565e-05 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.937491+0000", 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3975e-05 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.937539+0000", 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8798000000000003e-05 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.937553+0000", 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.401e-05 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.938612+0000", 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001058402 2024-09-13T11:32:27.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.938644+0000", 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1888999999999999e-05 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:158 4.4 e38/31 4:2b094039:::smithi20438004-40:head v 38'11, mlcod=38'9)", 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.943135+0000", 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.48620196399999999, 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002061939, 2024-09-13T11:32:27.643 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943135+0000", 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943135+0000", 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943136+0000", 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0400000000000005e-07 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943608+0000", 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047201399999999998 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943618+0000", 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7729999999999994e-06 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943624+0000", 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0390000000000001e-06 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943655+0000", 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1931000000000003e-05 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.943667+0000", 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1596999999999999e-05 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.945171+0000", 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001503594 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.945197+0000", 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6186999999999998e-05 2024-09-13T11:32:27.647 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.097084+0000", 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.33225320200000003, 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010008687, 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.648 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097084+0000", 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097084+0000", 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097084+0000", 2024-09-13T11:32:27.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9499999999999998e-07 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103199+0000", 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061146200000000003 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103222+0000", 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3176999999999998e-05 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103235+0000", 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2982999999999999e-05 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103317+0000", 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2412999999999999e-05 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103351+0000", 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3643000000000003e-05 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107033+0000", 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036821279999999998 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107092+0000", 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9327999999999998e-05 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:27.652 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.103265+0000", 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.326071836, 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012102717000000001, 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103265+0000", 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103265+0000", 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.103266+0000", 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1349999999999999e-06 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107571+0000", 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043046059999999999 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107584+0000", 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3519999999999999e-05 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107592+0000", 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0639999999999994e-06 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107665+0000", 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2234e-05 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107688+0000", 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3458000000000001e-05 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115305+0000", 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0076166059999999997 2024-09-13T11:32:27.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115368+0000", 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3094000000000003e-05 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.109354+0000", 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.31998265399999998, 2024-09-13T11:32:27.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096331119999999992, 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109354+0000", 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109354+0000", 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109355+0000", 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.26e-06 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.113722+0000", 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043666709999999999 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.113732+0000", 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9739999999999997e-06 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.113747+0000", 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4396000000000001e-05 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.113805+0000", 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8130000000000001e-05 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.113823+0000", 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8661999999999999e-05 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.118928+0000", 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051044920000000004 2024-09-13T11:32:27.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.118987+0000", 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9527000000000001e-05 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:224 3.3 e38/30 3:e7e3acce:::smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=38'1)", 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.144023+0000", 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.28531346200000002, 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014703733, 2024-09-13T11:32:27.662 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144023+0000", 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144023+0000", 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144025+0000", 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2419999999999999e-06 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148581+0000", 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004555992 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148595+0000", 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4353e-05 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148602+0000", 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9410000000000001e-06 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148656+0000", 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4455e-05 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148683+0000", 2024-09-13T11:32:27.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6319999999999999e-05 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.158655+0000", 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099724519999999997 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.158727+0000", 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1977999999999994e-05 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.148619+0000", 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.28071765500000001, 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010919000999999999, 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148619+0000", 2024-09-13T11:32:27.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148619+0000", 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.148620+0000", 2024-09-13T11:32:27.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1900000000000003e-07 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155189+0000", 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065693219999999998 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155203+0000", 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4363e-05 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155212+0000", 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7560000000000005e-06 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155272+0000", 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0124000000000002e-05 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155298+0000", 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5754000000000001e-05 2024-09-13T11:32:27.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.159478+0000", 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041801590000000001 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.159538+0000", 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0003999999999997e-05 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.27410011200000001, 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016373579999999999, 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155237+0000", 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0100000000000005e-07 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164267+0000", 2024-09-13T11:32:27.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090294950000000002 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164286+0000", 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9663000000000001e-05 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164298+0000", 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1813e-05 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164398+0000", 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9739000000000002e-05 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164430+0000", 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2458e-05 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.171556+0000", 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007125981 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.171610+0000", 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.393e-05 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.164342+0000", 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.26499485499999997, 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023411006000000002, 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.676 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164342+0000", 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164342+0000", 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.164343+0000", 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3099999999999997e-07 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178702+0000", 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014358963000000001 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178724+0000", 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1929999999999998e-05 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178739+0000", 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5747999999999999e-05 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178818+0000", 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8969999999999998e-05 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178845+0000", 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6804999999999998e-05 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.187698+0000", 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088531540000000002 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.187753+0000", 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4805000000000001e-05 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.178775+0000", 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.25056191100000003, 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097399329999999992, 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178775+0000", 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178775+0000", 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.178776+0000", 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.11e-06 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183714+0000", 2024-09-13T11:32:27.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049379599999999999 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183727+0000", 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2456999999999999e-05 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183735+0000", 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4840000000000004e-06 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183799+0000", 2024-09-13T11:32:27.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3596999999999994e-05 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183826+0000", 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7382e-05 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.188464+0000", 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004638344 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.188515+0000", 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0599000000000001e-05 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.183763+0000", 2024-09-13T11:32:27.685 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.24557425899999999, 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090443720000000002, 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183763+0000", 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183763+0000", 2024-09-13T11:32:27.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.183763+0000", 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6000000000000003e-07 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.189940+0000", 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006177035 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.189952+0000", 2024-09-13T11:32:27.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1838000000000001e-05 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.189963+0000", 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1139e-05 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.190017+0000", 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3758999999999998e-05 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.190041+0000", 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4131e-05 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.192768+0000", 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027266999999999999 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.192807+0000", 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9110000000000003e-05 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.689 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.212947+0000", 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.21638947, 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010976932999999999, 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.212947+0000", 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.212947+0000", 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.212949+0000", 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3629999999999999e-06 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220687+0000", 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077381810000000002 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220706+0000", 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9383000000000001e-05 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220716+0000", 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6849999999999996e-06 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220783+0000", 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7002000000000007e-05 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220807+0000", 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4128999999999999e-05 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223869+0000", 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00306143 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223924+0000", 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5760000000000001e-05 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.3.22:1 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.221550+0000", 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.20778726, 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025308790000000002, 2024-09-13T11:32:27.694 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221550+0000", 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221550+0000", 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221551+0000", 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1039999999999999e-06 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221557+0000", 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8019999999999997e-06 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221569+0000", 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1997999999999999e-05 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221576+0000", 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9789999999999996e-06 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222608+0000", 2024-09-13T11:32:27.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001032144 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222624+0000", 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6291e-05 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224045+0000", 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014212249999999999 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224081+0000", 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5336000000000002e-05 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.1.22:3 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.222261+0000", 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.207076343, 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001847163, 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.699 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222261+0000", 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222261+0000", 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222262+0000", 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.065e-06 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222271+0000", 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4919999999999999e-06 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222284+0000", 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2904999999999999e-05 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222291+0000", 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4390000000000003e-06 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222812+0000", 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00052035099999999995 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222827+0000", 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4901e-05 2024-09-13T11:32:27.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224083+0000", 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012563330000000001 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224108+0000", 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4677000000000001e-05 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.12.22:16 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'17, mlcod=38'12)", 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.233006+0000", 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.19633050199999999, 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034814289999999999, 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233006+0000", 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233006+0000", 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233007+0000", 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4799999999999997e-07 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233013+0000", 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7389999999999996e-06 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233025+0000", 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2520999999999999e-05 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233037+0000", 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1770000000000001e-05 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233129+0000", 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1574000000000001e-05 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233148+0000", 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9171999999999999e-05 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236437+0000", 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003289312 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236488+0000", 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0593000000000002e-05 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:271 4.4 e38/31 4:2c957d44:::smithi20438004-22:head v 38'15, mlcod=38'13)", 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.233290+0000", 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.19604661300000001, 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031153359999999998, 2024-09-13T11:32:27.708 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233290+0000", 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233290+0000", 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233291+0000", 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5400000000000001e-07 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233296+0000", 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1109999999999999e-06 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233308+0000", 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1783999999999999e-05 2024-09-13T11:32:27.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233314+0000", 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3459999999999999e-06 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233365+0000", 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0853000000000003e-05 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233385+0000", 2024-09-13T11:32:27.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0083000000000001e-05 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236368+0000", 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029832970000000002 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236406+0000", 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7308000000000003e-05 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:772 3.3 e38/30 3:dd3ad6bf:::4ec2dd001b7153a0bb00c76a8320dec150c3eb87238ccdd9135be5e948aee2d9:head v 38'30, mlcod=38'28)", 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.359524+0000", 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.069812727000000005, 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061240950000000004, 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.359524+0000", 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.359524+0000", 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.359526+0000", 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.361515+0000", 2024-09-13T11:32:27.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001989532 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.361533+0000", 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7581999999999999e-05 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.361542+0000", 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5170000000000005e-06 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:27.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.361600+0000", 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7927999999999998e-05 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.361619+0000", 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8892000000000001e-05 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.365604+0000", 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003984792 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:27.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.365648+0000", 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4425000000000002e-05 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:27.717 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:27.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:32:27.718 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:27.617+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:27.719 INFO:tasks.rados.rados.0.smithi204.stdout:610: writing smithi20438004-38 from 457157 to 992974 tid 1 2024-09-13T11:32:27.719 INFO:tasks.rados.rados.0.smithi204.stdout:610: writing smithi20438004-38 from 1768111 to 2170788 tid 2 2024-09-13T11:32:27.719 INFO:tasks.rados.rados.0.smithi204.stdout:611: read oid 44 snap -1 2024-09-13T11:32:27.719 INFO:tasks.rados.rados.0.smithi204.stdout:611: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:612: tier_promote oid 6 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:603: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:601: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:602: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:604: finishing set_chunk to oid 25 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 995840 length: 239616 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:605: finishing write tid 1 to smithi20438004-50 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:605: finishing write tid 2 to smithi20438004-50 2024-09-13T11:32:27.720 INFO:tasks.rados.rados.0.smithi204.stdout:605: finishing write tid 3 to smithi20438004-50 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:605: finishing write tid 4 to smithi20438004-50 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:605: finishing write tid 5 to smithi20438004-50 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:605: oid 50 updating version 0 to 19 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:605: oid 50 version 19 is already newer than 17 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:605: oid 50 version 19 is already newer than 18 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:605: oid 50 version 19 is already newer than 16 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:605: left oid 50 (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:606: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 126591 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:607: finishing set_chunk to oid 29 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 1014784 length: 1380864 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:608: finishing set_chunk to oid 17 2024-09-13T11:32:27.721 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk returned error code -95 offset: 463872 length: 642048 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:609: finishing set_chunk to oid 27 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 1058304 length: 285696 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:610: finishing write tid 1 to smithi20438004-38 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:610: finishing write tid 2 to smithi20438004-38 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:601: done (11 left) 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:602: done (10 left) 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:603: done (9 left) 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:604: done (8 left) 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:605: done (7 left) 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:606: done (6 left) 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:607: done (5 left) 2024-09-13T11:32:27.722 INFO:tasks.rados.rados.0.smithi204.stdout:608: done (4 left) 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:609: done (3 left) 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:613: read oid 28 snap -1 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:613: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:614: tier_flush oid 47 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:614: tier_flush oid 47 snap -1 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:610: finishing write tid 3 to smithi20438004-38 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:610: finishing write tid 4 to smithi20438004-38 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:610: oid 38 updating version 0 to 32 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:610: oid 38 updating version 32 to 33 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:610: oid 38 version 33 is already newer than 31 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 33 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:610: left oid 38 (ObjNum 51 snap 0 seq_num 51) 2024-09-13T11:32:27.723 INFO:tasks.rados.rados.0.smithi204.stdout:611: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:27.724 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:27.723+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:27.732 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:27.791 INFO:tasks.rados.rados.0.smithi204.stdout:612: got (0) Success 2024-09-13T11:32:27.791 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:27.791 INFO:tasks.rados.rados.0.smithi204.stdout:610: done (4 left) 2024-09-13T11:32:27.791 INFO:tasks.rados.rados.0.smithi204.stdout:611: done (3 left) 2024-09-13T11:32:27.791 INFO:tasks.rados.rados.0.smithi204.stdout:612: done (2 left) 2024-09-13T11:32:27.791 INFO:tasks.rados.rados.0.smithi204.stdout:615: read oid 21 snap -1 2024-09-13T11:32:27.791 INFO:tasks.rados.rados.0.smithi204.stdout:615: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T11:32:27.792 INFO:tasks.rados.rados.0.smithi204.stdout:616: write oid 7 current snap is 0 2024-09-13T11:32:27.792 INFO:tasks.rados.rados.0.smithi204.stdout:616: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-09-13T11:32:27.796 INFO:tasks.rados.rados.0.smithi204.stdout:616: writing smithi20438004-7 from 473964 to 1085030 tid 1 2024-09-13T11:32:27.802 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:27.804 INFO:tasks.rados.rados.0.smithi204.stdout:616: writing smithi20438004-7 from 1826593 to 2372928 tid 2 2024-09-13T11:32:27.807 INFO:tasks.rados.rados.0.smithi204.stdout:616: writing smithi20438004-7 from 2778819 to 2873964 tid 3 2024-09-13T11:32:27.807 INFO:tasks.rados.rados.0.smithi204.stdout:617: read oid 38 snap -1 2024-09-13T11:32:27.807 INFO:tasks.rados.rados.0.smithi204.stdout:617: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-13T11:32:27.807 INFO:tasks.rados.rados.0.smithi204.stdout:618: read oid 45 snap -1 2024-09-13T11:32:27.807 INFO:tasks.rados.rados.0.smithi204.stdout:618: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-13T11:32:27.807 INFO:tasks.rados.rados.0.smithi204.stdout:619: read oid 18 snap -1 2024-09-13T11:32:27.807 INFO:tasks.rados.rados.0.smithi204.stdout:619: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T11:32:27.808 INFO:tasks.rados.rados.0.smithi204.stdout:613: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-13T11:32:27.826 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:27.824+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:27.829 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:27.829 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:27.829 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:27.829 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:27.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:32:27.863 INFO:tasks.rados.rados.0.smithi204.stdout:614: got (0) Success 2024-09-13T11:32:27.863 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-13T11:32:27.863 INFO:tasks.rados.rados.0.smithi204.stdout:620: read oid 33 snap -1 2024-09-13T11:32:27.863 INFO:tasks.rados.rados.0.smithi204.stdout:620: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T11:32:27.863 INFO:tasks.rados.rados.0.smithi204.stdout:613: done (7 left) 2024-09-13T11:32:27.863 INFO:tasks.rados.rados.0.smithi204.stdout:614: done (6 left) 2024-09-13T11:32:27.863 INFO:tasks.rados.rados.0.smithi204.stdout:621: set_chunk oid 40 target oid 2024-09-13T11:32:27.867 INFO:tasks.rados.rados.0.smithi204.stdout:621: set_chunk oid 40 offset: 1888768 length: 233472 target oid eabd0077379227263786827fb82b287d0ef772bb9542008c45eca61a0ce6a8f0 offset: 0 snap -1 2024-09-13T11:32:27.867 INFO:tasks.rados.rados.0.smithi204.stdout:622: read oid 35 snap -1 2024-09-13T11:32:27.867 INFO:tasks.rados.rados.0.smithi204.stdout:622: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T11:32:27.867 INFO:tasks.rados.rados.0.smithi204.stdout:623: read oid 15 snap -1 2024-09-13T11:32:27.867 INFO:tasks.rados.rados.0.smithi204.stdout:623: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T11:32:27.867 INFO:tasks.rados.rados.0.smithi204.stdout:624: set_chunk oid 10 target oid 2024-09-13T11:32:27.885 INFO:tasks.rados.rados.0.smithi204.stdout:624: set_chunk oid 10 offset: 666624 length: 353280 target oid bc036ea868a6b2c601148e4e87101fdc0cdd45a73131aa4e7533f4932f45b24d offset: 0 snap -1 2024-09-13T11:32:27.885 INFO:tasks.rados.rados.0.smithi204.stdout:625: set_chunk oid 25 target oid 2024-09-13T11:32:27.900 INFO:tasks.rados.rados.0.smithi204.stdout:625: set_chunk oid 25 offset: 169984 length: 1435136 target oid 748976ac7fbeb0c7d36e3ea1bc03d9de6be9fd7a451e97d4acfdce8738477adf offset: 0 snap -1 2024-09-13T11:32:27.901 INFO:tasks.rados.rados.0.smithi204.stdout:626: set_chunk oid 16 target oid 2024-09-13T11:32:27.917 INFO:tasks.rados.rados.0.smithi204.stdout:626: set_chunk oid 16 offset: 168448 length: 1795584 target oid 01188d4bc792dd183d38042412b10b832568a77fbaab8a56c6a16d0e15e28c37 offset: 0 snap -1 2024-09-13T11:32:27.917 INFO:tasks.rados.rados.0.smithi204.stdout:627: tier_evict oid 2 2024-09-13T11:32:27.917 INFO:tasks.rados.rados.0.smithi204.stdout:627: tier_evict oid 2 snap -1 2024-09-13T11:32:27.917 INFO:tasks.rados.rados.0.smithi204.stdout:615: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T11:32:27.926 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:27.925+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:27.941 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:27.944 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:27.962 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:27.962 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:27.967 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:27.978 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-09-13T11:32:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:32:27.981 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:27.981 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:27.981 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:27.981 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:27.982 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:27.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:32:27.994 INFO:tasks.rados.rados.0.smithi204.stdout:616: finishing write tid 1 to smithi20438004-7 2024-09-13T11:32:27.994 INFO:tasks.rados.rados.0.smithi204.stdout:616: finishing write tid 2 to smithi20438004-7 2024-09-13T11:32:27.994 INFO:tasks.rados.rados.0.smithi204.stdout:617: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-13T11:32:27.995 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:28.011 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:28.012 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:28.027 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:28.026+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:28.028 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:28.028 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:28.049 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:28.049 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:28.063 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:28.078 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:28.079 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:28.093 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.094 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.095 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:28.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:32:28.113 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:28.113 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:28.116 INFO:tasks.rados.rados.0.smithi204.stdout:619: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-13T11:32:28.116 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:28.128 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:28.127+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:28.129 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:28.129 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:28.145 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:28.146 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:28.162 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:28.162 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:28.179 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:28.179 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:28.190 INFO:tasks.rados.rados.0.smithi204.stdout:618: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-13T11:32:28.190 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:28.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:28.194+0000 7fdaf359e640 0 -- 172.21.15.38:0/3457908793 >> v2:172.21.15.204:6812/501019151 conn(0x7fdaf4222aa0 msgr2=0x7fdaf4224e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:28.197 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:28.208 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.208 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.216 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:32:28.220 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:28.229 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:28.227+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:28.251 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:28.281 INFO:tasks.rados.rados.0.smithi204.stdout:616: finishing write tid 3 to smithi20438004-7 2024-09-13T11:32:28.281 INFO:tasks.rados.rados.0.smithi204.stdout:616: finishing write tid 4 to smithi20438004-7 2024-09-13T11:32:28.281 INFO:tasks.rados.rados.0.smithi204.stdout:616: finishing write tid 5 to smithi20438004-7 2024-09-13T11:32:28.281 INFO:tasks.rados.rados.0.smithi204.stdout:616: oid 7 updating version 0 to 20 2024-09-13T11:32:28.281 INFO:tasks.rados.rados.0.smithi204.stdout:616: oid 7 version 20 is already newer than 19 2024-09-13T11:32:28.281 INFO:tasks.rados.rados.0.smithi204.stdout:616: oid 7 version 20 is already newer than 18 2024-09-13T11:32:28.281 INFO:tasks.rados.rados.0.smithi204.stdout:616: oid 7 version 20 is already newer than 17 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 20 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:616: left oid 7 (ObjNum 52 snap 0 seq_num 52) 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:615: done (12 left) 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:616: done (11 left) 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:617: done (10 left) 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:618: done (9 left) 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:619: done (8 left) 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:628: delete oid 19 current snap is 0 2024-09-13T11:32:28.282 INFO:tasks.rados.rados.0.smithi204.stdout:620: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-13T11:32:28.283 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1045371 2024-09-13T11:32:28.320 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.321 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:32:28.330 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:28.328+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:28.345 INFO:tasks.rados.rados.0.smithi204.stdout:621: finishing set_chunk to oid 40 2024-09-13T11:32:28.345 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 40 set_chunk returned error code -95 offset: 1888768 length: 233472 2024-09-13T11:32:28.346 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-13T11:32:28.346 INFO:tasks.rados.rados.0.smithi204.stdout:623: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-13T11:32:28.346 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:28.374 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 797878 2024-09-13T11:32:28.393 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1874288 2024-09-13T11:32:28.403 INFO:tasks.rados.rados.0.smithi204.stdout:622: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-13T11:32:28.430 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:28.429+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:28.433 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.434 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:32:28.484 INFO:tasks.rados.rados.0.smithi204.stdout:624: finishing set_chunk to oid 10 2024-09-13T11:32:28.484 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 666624 length: 353280 2024-09-13T11:32:28.484 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-13T11:32:28.484 INFO:tasks.rados.rados.0.smithi204.stdout:625: finishing set_chunk to oid 25 2024-09-13T11:32:28.484 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 169984 length: 1435136 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:627: got (22) Invalid argument 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:626: finishing set_chunk to oid 16 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 168448 length: 1795584 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:620: done (8 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:621: done (7 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:622: done (6 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:623: done (5 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:624: done (4 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:625: done (3 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:626: done (2 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:627: done (1 left) 2024-09-13T11:32:28.485 INFO:tasks.rados.rados.0.smithi204.stdout:628: done (0 left) 2024-09-13T11:32:28.486 INFO:tasks.rados.rados.0.smithi204.stdout:629: write oid 17 current snap is 0 2024-09-13T11:32:28.486 INFO:tasks.rados.rados.0.smithi204.stdout:629: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-09-13T11:32:28.489 INFO:tasks.rados.rados.0.smithi204.stdout:629: writing smithi20438004-17 from 507578 to 985495 tid 1 2024-09-13T11:32:28.495 INFO:tasks.rados.rados.0.smithi204.stdout:629: writing smithi20438004-17 from 1543557 to 2093561 tid 2 2024-09-13T11:32:28.499 INFO:tasks.rados.rados.0.smithi204.stdout:629: writing smithi20438004-17 from 2833665 to 2907578 tid 3 2024-09-13T11:32:28.499 INFO:tasks.rados.rados.0.smithi204.stdout:630: set_chunk oid 5 target oid 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:630: set_chunk oid 5 offset: 314880 length: 1546240 target oid 194f870459df717b3a43948c3b56bc35cac5308afcac3cd76a1a7e1a04723236 offset: 0 snap -1 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:631: read oid 26 snap -1 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:631: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:632: write oid 36 current snap is 0 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:629: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:629: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:629: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:629: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:629: finishing write tid 5 to smithi20438004-17 2024-09-13T11:32:28.515 INFO:tasks.rados.rados.0.smithi204.stdout:629: oid 17 updating version 0 to 20 2024-09-13T11:32:28.516 INFO:tasks.rados.rados.0.smithi204.stdout:629: oid 17 version 20 is already newer than 18 2024-09-13T11:32:28.516 INFO:tasks.rados.rados.0.smithi204.stdout:629: oid 17 updating version 20 to 21 2024-09-13T11:32:28.516 INFO:tasks.rados.rados.0.smithi204.stdout:629: oid 17 version 21 is already newer than 19 2024-09-13T11:32:28.516 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 21 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-09-13T11:32:28.516 INFO:tasks.rados.rados.0.smithi204.stdout:629: left oid 17 (ObjNum 54 snap 0 seq_num 54) 2024-09-13T11:32:28.516 INFO:tasks.rados.rados.0.smithi204.stdout:632: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-13T11:32:28.520 INFO:tasks.rados.rados.0.smithi204.stdout:632: writing smithi20438004-36 from 524385 to 1077551 tid 1 2024-09-13T11:32:28.525 INFO:tasks.rados.rados.0.smithi204.stdout:632: writing smithi20438004-36 from 1602039 to 2012054 tid 2 2024-09-13T11:32:28.531 INFO:tasks.rados.rados.0.smithi204.stdout:632: writing smithi20438004-36 from 2577441 to 2924385 tid 3 2024-09-13T11:32:28.531 INFO:tasks.rados.rados.0.smithi204.stdout:629: done (3 left) 2024-09-13T11:32:28.531 INFO:tasks.rados.rados.0.smithi204.stdout:633: write (excl) oid 15 current snap is 0 2024-09-13T11:32:28.531 INFO:tasks.rados.rados.0.smithi204.stdout:630: finishing set_chunk to oid 5 2024-09-13T11:32:28.531 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 314880 length: 1546240 2024-09-13T11:32:28.531 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-13T11:32:28.531 INFO:tasks.rados.rados.0.smithi204.stdout:631: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-13T11:32:28.531 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:28.529+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:28.551 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.551 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:32:28.577 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 417908 2024-09-13T11:32:28.600 INFO:tasks.rados.rados.0.smithi204.stdout:632: finishing write tid 1 to smithi20438004-36 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: finishing write tid 2 to smithi20438004-36 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: finishing write tid 3 to smithi20438004-36 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: finishing write tid 4 to smithi20438004-36 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: finishing write tid 5 to smithi20438004-36 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: oid 36 updating version 0 to 19 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: oid 36 version 19 is already newer than 16 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: oid 36 version 19 is already newer than 18 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: oid 36 version 19 is already newer than 17 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 19 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:632: left oid 36 (ObjNum 55 snap 0 seq_num 55) 2024-09-13T11:32:28.601 INFO:tasks.rados.rados.0.smithi204.stdout:633: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-13T11:32:28.606 INFO:tasks.rados.rados.0.smithi204.stdout:633: writing smithi20438004-15 from 541192 to 1169607 tid 1 2024-09-13T11:32:28.612 INFO:tasks.rados.rados.0.smithi204.stdout:633: writing smithi20438004-15 from 1660521 to 2214194 tid 2 2024-09-13T11:32:28.616 INFO:tasks.rados.rados.0.smithi204.stdout:633: writing smithi20438004-15 from 2888511 to 2941192 tid 3 2024-09-13T11:32:28.617 INFO:tasks.rados.rados.0.smithi204.stdout:630: done (3 left) 2024-09-13T11:32:28.617 INFO:tasks.rados.rados.0.smithi204.stdout:631: done (2 left) 2024-09-13T11:32:28.617 INFO:tasks.rados.rados.0.smithi204.stdout:632: done (1 left) 2024-09-13T11:32:28.617 INFO:tasks.rados.rados.0.smithi204.stdout:634: write (excl) oid 39 current snap is 0 2024-09-13T11:32:28.617 INFO:tasks.rados.rados.0.smithi204.stdout:633: finishing write tid 1 to smithi20438004-15 2024-09-13T11:32:28.617 INFO:tasks.rados.rados.0.smithi204.stdout:634: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-13T11:32:28.623 INFO:tasks.rados.rados.0.smithi204.stdout:634: writing smithi20438004-39 from 557999 to 1261663 tid 1 2024-09-13T11:32:28.628 INFO:tasks.rados.rados.0.smithi204.stdout:634: writing smithi20438004-39 from 1719003 to 2132687 tid 2 2024-09-13T11:32:28.631 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:28.629+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:28.632 INFO:tasks.rados.rados.0.smithi204.stdout:634: writing smithi20438004-39 from 2632287 to 2957999 tid 3 2024-09-13T11:32:28.632 INFO:tasks.rados.rados.0.smithi204.stdout:635: read oid 7 snap -1 2024-09-13T11:32:28.633 INFO:tasks.rados.rados.0.smithi204.stdout:635: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-13T11:32:28.633 INFO:tasks.rados.rados.0.smithi204.stdout:636: set_chunk oid 3 target oid 2024-09-13T11:32:28.633 INFO:tasks.rados.rados.0.smithi204.stdout:633: finishing write tid 2 to smithi20438004-15 2024-09-13T11:32:28.633 INFO:tasks.rados.rados.0.smithi204.stdout:633: finishing write tid 3 to smithi20438004-15 2024-09-13T11:32:28.633 INFO:tasks.rados.rados.0.smithi204.stdout:633: finishing write tid 4 to smithi20438004-15 2024-09-13T11:32:28.644 INFO:tasks.rados.rados.0.smithi204.stdout:636: set_chunk oid 3 offset: 559104 length: 812544 target oid 264b891cbac90c881a529a0f2044e8ee33f56ba1d585376ff23a4a5f3c4b77da offset: 0 snap -1 2024-09-13T11:32:28.644 INFO:tasks.rados.rados.0.smithi204.stdout:637: set_chunk oid 18 target oid 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:637: set_chunk oid 18 offset: 266752 length: 71680 target oid 69d03b1809e87b2b41cf3a4e8af1771305bca1534249119f5a1a7cb2e90a1b9a offset: 0 snap -1 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:638: write oid 27 current snap is 0 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:633: finishing write tid 5 to smithi20438004-15 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:633: oid 15 updating version 0 to 23 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:633: oid 15 updating version 23 to 24 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:633: oid 15 updating version 24 to 25 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:633: oid 15 version 25 is already newer than 22 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 25 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-09-13T11:32:28.647 INFO:tasks.rados.rados.0.smithi204.stdout:633: left oid 15 (ObjNum 56 snap 0 seq_num 56) 2024-09-13T11:32:28.648 INFO:tasks.rados.rados.0.smithi204.stdout:634: finishing write tid 1 to smithi20438004-39 2024-09-13T11:32:28.648 INFO:tasks.rados.rados.0.smithi204.stdout:634: finishing write tid 2 to smithi20438004-39 2024-09-13T11:32:28.648 INFO:tasks.rados.rados.0.smithi204.stdout:638: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-13T11:32:28.653 INFO:tasks.rados.rados.0.smithi204.stdout:638: writing smithi20438004-27 from 574806 to 1353719 tid 1 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:638: writing smithi20438004-27 from 2061132 to 2618474 tid 2 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:633: done (5 left) 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:639: tier_promote oid 26 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:634: finishing write tid 3 to smithi20438004-39 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:634: finishing write tid 4 to smithi20438004-39 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:634: finishing write tid 5 to smithi20438004-39 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:634: oid 39 updating version 0 to 25 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:634: oid 39 version 25 is already newer than 24 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:634: oid 39 updating version 25 to 26 2024-09-13T11:32:28.661 INFO:tasks.rados.rados.0.smithi204.stdout:634: oid 39 version 26 is already newer than 23 2024-09-13T11:32:28.662 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 26 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-13T11:32:28.662 INFO:tasks.rados.rados.0.smithi204.stdout:634: left oid 39 (ObjNum 57 snap 0 seq_num 57) 2024-09-13T11:32:28.662 INFO:tasks.rados.rados.0.smithi204.stdout:635: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-13T11:32:28.670 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.670 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:32:28.707 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:28.763 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:28.762+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:28.778 INFO:tasks.rados.rados.0.smithi204.stdout:636: finishing set_chunk to oid 3 2024-09-13T11:32:28.778 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 559104 length: 812544 2024-09-13T11:32:28.778 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:28.778 INFO:tasks.rados.rados.0.smithi204.stdout:637: finishing set_chunk to oid 18 2024-09-13T11:32:28.778 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk returned error code -95 offset: 266752 length: 71680 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:638: finishing write tid 1 to smithi20438004-27 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:638: finishing write tid 2 to smithi20438004-27 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:634: done (5 left) 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:635: done (4 left) 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:636: done (3 left) 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:637: done (2 left) 2024-09-13T11:32:28.779 INFO:tasks.rados.rados.0.smithi204.stdout:640: set_chunk oid 36 target oid 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:640: set_chunk oid 36 offset: 698368 length: 1183744 target oid e7822a0af4929b07914c95bd62b80a07f8c2bcd9ee6befe6bd0644e438a93e7e offset: 0 snap -1 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:641: write (excl) oid 25 current snap is 0 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:638: finishing write tid 3 to smithi20438004-27 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:638: finishing write tid 4 to smithi20438004-27 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:638: oid 27 updating version 0 to 28 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:638: oid 27 updating version 28 to 29 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:638: oid 27 version 29 is already newer than 27 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 29 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:638: left oid 27 (ObjNum 58 snap 0 seq_num 58) 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:639: got (0) Success 2024-09-13T11:32:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-13T11:32:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:641: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-13T11:32:28.798 INFO:tasks.rados.rados.0.smithi204.stdout:641: writing smithi20438004-25 from 591613 to 1045775 tid 1 2024-09-13T11:32:28.804 INFO:tasks.rados.rados.0.smithi204.stdout:641: writing smithi20438004-25 from 1719614 to 2136967 tid 2 2024-09-13T11:32:28.809 INFO:tasks.rados.rados.0.smithi204.stdout:641: writing smithi20438004-25 from 2570780 to 2991613 tid 3 2024-09-13T11:32:28.809 INFO:tasks.rados.rados.0.smithi204.stdout:638: done (3 left) 2024-09-13T11:32:28.809 INFO:tasks.rados.rados.0.smithi204.stdout:639: done (2 left) 2024-09-13T11:32:28.809 INFO:tasks.rados.rados.0.smithi204.stdout:642: set_chunk oid 19 target oid 2024-09-13T11:32:28.809 INFO:tasks.rados.rados.0.smithi204.stdout:642: set_chunk oid 19 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:28.810 INFO:tasks.rados.rados.0.smithi204.stdout:643: read oid 5 snap -1 2024-09-13T11:32:28.810 INFO:tasks.rados.rados.0.smithi204.stdout:643: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T11:32:28.810 INFO:tasks.rados.rados.0.smithi204.stdout:644: delete oid 28 current snap is 0 2024-09-13T11:32:28.810 INFO:tasks.rados.rados.0.smithi204.stdout:640: finishing set_chunk to oid 36 2024-09-13T11:32:28.810 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 19 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-13T11:32:28.810 INFO:tasks.rados.rados.0.smithi204.stdout:641: finishing write tid 1 to smithi20438004-25 2024-09-13T11:32:28.810 INFO:tasks.rados.rados.0.smithi204.stdout:641: finishing write tid 2 to smithi20438004-25 2024-09-13T11:32:28.814 INFO:tasks.rados.rados.0.smithi204.stdout:641: finishing write tid 3 to smithi20438004-25 2024-09-13T11:32:28.815 INFO:tasks.rados.rados.0.smithi204.stdout:643: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-13T11:32:28.815 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:28.828 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.828 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.832 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:28.831+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:28.837 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:32:28.873 INFO:tasks.rados.rados.0.smithi204.stdout:642: finishing set_chunk to oid 19 2024-09-13T11:32:28.873 INFO:tasks.rados.rados.0.smithi204.stdout:642: got expected ENOENT (src dne) 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: finishing write tid 4 to smithi20438004-25 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: finishing write tid 5 to smithi20438004-25 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: oid 25 updating version 0 to 33 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: oid 25 version 33 is already newer than 31 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: oid 25 version 33 is already newer than 30 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: oid 25 version 33 is already newer than 32 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 33 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: left oid 25 (ObjNum 59 snap 0 seq_num 59) 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:640: done (4 left) 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:641: done (3 left) 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:642: done (2 left) 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:643: done (1 left) 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:644: done (0 left) 2024-09-13T11:32:28.874 INFO:tasks.rados.rados.0.smithi204.stdout:645: write oid 47 current snap is 0 2024-09-13T11:32:28.875 INFO:tasks.rados.rados.0.smithi204.stdout:645: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-13T11:32:28.880 INFO:tasks.rados.rados.0.smithi204.stdout:645: writing smithi20438004-47 from 625227 to 1346240 tid 1 2024-09-13T11:32:28.888 INFO:tasks.rados.rados.0.smithi204.stdout:645: writing smithi20438004-47 from 1952931 to 2657600 tid 2 2024-09-13T11:32:28.888 INFO:tasks.rados.rados.0.smithi204.stdout:646: write (excl) oid 23 current snap is 0 2024-09-13T11:32:28.888 INFO:tasks.rados.rados.0.smithi204.stdout:645: finishing write tid 1 to smithi20438004-47 2024-09-13T11:32:28.888 INFO:tasks.rados.rados.0.smithi204.stdout:646: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-13T11:32:28.896 INFO:tasks.rados.rados.0.smithi204.stdout:646: writing smithi20438004-23 from 642034 to 1438296 tid 1 2024-09-13T11:32:28.902 INFO:tasks.rados.rados.0.smithi204.stdout:646: writing smithi20438004-23 from 1895060 to 2459740 tid 2 2024-09-13T11:32:28.902 INFO:tasks.rados.rados.0.smithi204.stdout:647: write (excl) oid 31 current snap is 0 2024-09-13T11:32:28.902 INFO:tasks.rados.rados.0.smithi204.stdout:645: finishing write tid 2 to smithi20438004-47 2024-09-13T11:32:28.902 INFO:tasks.rados.rados.0.smithi204.stdout:647: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-13T11:32:28.908 INFO:tasks.rados.rados.0.smithi204.stdout:647: writing smithi20438004-31 from 658841 to 1130352 tid 1 2024-09-13T11:32:28.915 INFO:tasks.rados.rados.0.smithi204.stdout:647: writing smithi20438004-31 from 1553542 to 2261880 tid 2 2024-09-13T11:32:28.920 INFO:tasks.rados.rados.0.smithi204.stdout:647: writing smithi20438004-31 from 2964119 to 3058841 tid 3 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:648: read oid 37 snap -1 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:648: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:649: tier_promote oid 50 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:645: finishing write tid 3 to smithi20438004-47 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:645: finishing write tid 4 to smithi20438004-47 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:645: oid 47 updating version 0 to 14 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:645: oid 47 version 14 is already newer than 13 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:645: oid 47 updating version 14 to 15 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 15 (ObjNum 61 snap 0 seq_num 61) dirty exists 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:645: left oid 47 (ObjNum 61 snap 0 seq_num 61) 2024-09-13T11:32:28.921 INFO:tasks.rados.rados.0.smithi204.stdout:645: done (4 left) 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:650: read oid 17 snap -1 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:650: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:646: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:646: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:646: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:651: read oid 40 snap -1 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:651: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:652: tier_evict oid 11 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:652: tier_evict oid 11 snap -1 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:646: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:646: oid 23 updating version 0 to 22 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:646: oid 23 updating version 22 to 23 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:646: oid 23 version 23 is already newer than 21 2024-09-13T11:32:28.922 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 23 (ObjNum 62 snap 0 seq_num 62) dirty exists 2024-09-13T11:32:28.923 INFO:tasks.rados.rados.0.smithi204.stdout:646: left oid 23 (ObjNum 62 snap 0 seq_num 62) 2024-09-13T11:32:28.923 INFO:tasks.rados.rados.0.smithi204.stdout:646: done (6 left) 2024-09-13T11:32:28.923 INFO:tasks.rados.rados.0.smithi204.stdout:653: set_chunk oid 3 target oid 2024-09-13T11:32:28.934 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:28.933+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:653: set_chunk oid 3 offset: 227328 length: 1324032 target oid 5a396ad7289f370ae7206e31a908942d11d50e8be30ee2251a7d24131be37eb6 offset: 0 snap -1 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:654: read oid 20 snap -1 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:654: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:655: read oid 7 snap -1 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:655: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:656: write (excl) oid 5 current snap is 0 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:647: finishing write tid 1 to smithi20438004-31 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:647: finishing write tid 2 to smithi20438004-31 2024-09-13T11:32:28.935 INFO:tasks.rados.rados.0.smithi204.stdout:656: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-13T11:32:28.941 INFO:tasks.rados.rados.0.smithi204.stdout:656: writing smithi20438004-5 from 675648 to 1222408 tid 1 2024-09-13T11:32:28.949 INFO:tasks.rados.rados.0.smithi204.stdout:656: writing smithi20438004-5 from 2012024 to 2580373 tid 2 2024-09-13T11:32:28.949 INFO:tasks.rados.rados.0.smithi204.stdout:657: read oid 13 snap -1 2024-09-13T11:32:28.949 INFO:tasks.rados.rados.0.smithi204.stdout:657: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T11:32:28.949 INFO:tasks.rados.rados.0.smithi204.stdout:658: read oid 36 snap -1 2024-09-13T11:32:28.949 INFO:tasks.rados.rados.0.smithi204.stdout:658: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-13T11:32:28.949 INFO:tasks.rados.rados.0.smithi204.stdout:659: write oid 16 current snap is 0 2024-09-13T11:32:28.950 INFO:tasks.rados.rados.0.smithi204.stdout:652: got (22) Invalid argument 2024-09-13T11:32:28.950 INFO:tasks.rados.rados.0.smithi204.stdout:649: got (0) Success 2024-09-13T11:32:28.950 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-13T11:32:28.950 INFO:tasks.rados.rados.0.smithi204.stdout:647: finishing write tid 3 to smithi20438004-31 2024-09-13T11:32:28.950 INFO:tasks.rados.rados.0.smithi204.stdout:647: finishing write tid 4 to smithi20438004-31 2024-09-13T11:32:28.950 INFO:tasks.rados.rados.0.smithi204.stdout:659: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-13T11:32:28.950 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:28.950 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:28.956 INFO:tasks.rados.rados.0.smithi204.stdout:659: writing smithi20438004-16 from 692455 to 1314464 tid 1 2024-09-13T11:32:28.959 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:659: writing smithi20438004-16 from 2070506 to 2782513 tid 2 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:649: done (12 left) 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:652: done (11 left) 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:660: tier_flush oid 8 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:660: tier_flush oid 8 snap -1 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:647: finishing write tid 5 to smithi20438004-31 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:647: oid 31 updating version 0 to 16 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:647: oid 31 version 16 is already newer than 14 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:647: oid 31 version 16 is already newer than 15 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:647: oid 31 version 16 is already newer than 13 2024-09-13T11:32:28.965 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 16 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-09-13T11:32:28.966 INFO:tasks.rados.rados.0.smithi204.stdout:647: left oid 31 (ObjNum 63 snap 0 seq_num 63) 2024-09-13T11:32:28.966 INFO:tasks.rados.rados.0.smithi204.stdout:648: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-13T11:32:28.966 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.020 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.035 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:29.033+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:29.049 INFO:tasks.rados.rados.0.smithi204.stdout:653: finishing set_chunk to oid 3 2024-09-13T11:32:29.050 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 227328 length: 1324032 2024-09-13T11:32:29.050 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:29.050 INFO:tasks.rados.rados.0.smithi204.stdout:650: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-13T11:32:29.050 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.071 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.071 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.079 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:32:29.110 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1949231 2024-09-13T11:32:29.128 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.135 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:29.134+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:29.173 INFO:tasks.rados.rados.0.smithi204.stdout:655: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-13T11:32:29.183 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.183 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:32:29.235 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1884042 2024-09-13T11:32:29.236 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:29.234+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:29.282 INFO:tasks.rados.rados.0.smithi204.stdout:654: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:29.282 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.337 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 213430 2024-09-13T11:32:29.337 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.337 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.338 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:29.336+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:29.346 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:32:29.362 INFO:tasks.rados.rados.0.smithi204.stdout:651: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-13T11:32:29.369 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1169432 2024-09-13T11:32:29.389 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1819365 2024-09-13T11:32:29.435 INFO:tasks.rados.rados.0.smithi204.stdout:656: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:29.435 INFO:tasks.rados.rados.0.smithi204.stdout:656: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:29.435 INFO:tasks.rados.rados.0.smithi204.stdout:656: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:29.435 INFO:tasks.rados.rados.0.smithi204.stdout:647: done (11 left) 2024-09-13T11:32:29.435 INFO:tasks.rados.rados.0.smithi204.stdout:648: done (10 left) 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:650: done (9 left) 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:651: done (8 left) 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:653: done (7 left) 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:654: done (6 left) 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:655: done (5 left) 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:661: delete oid 22 current snap is 0 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:656: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:656: oid 5 updating version 0 to 22 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:656: oid 5 updating version 22 to 24 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:656: oid 5 version 24 is already newer than 23 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 24 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-13T11:32:29.436 INFO:tasks.rados.rados.0.smithi204.stdout:656: left oid 5 (ObjNum 64 snap 0 seq_num 64) 2024-09-13T11:32:29.437 INFO:tasks.rados.rados.0.smithi204.stdout:657: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T11:32:29.437 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.437 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:29.436+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:29.459 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.466 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.466 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.475 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:32:29.503 INFO:tasks.rados.rados.0.smithi204.stdout:658: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-13T11:32:29.538 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:29.537+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:29.570 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 367624 2024-09-13T11:32:29.607 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.608 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: finishing write tid 1 to smithi20438004-16 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: finishing write tid 2 to smithi20438004-16 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: finishing write tid 3 to smithi20438004-16 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: finishing write tid 4 to smithi20438004-16 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: oid 16 updating version 0 to 23 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: oid 16 version 23 is already newer than 22 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: oid 16 version 23 is already newer than 21 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 23 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:659: left oid 16 (ObjNum 65 snap 0 seq_num 65) 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:660: got (0) Success 2024-09-13T11:32:29.612 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:656: done (5 left) 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:657: done (4 left) 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:658: done (3 left) 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:659: done (2 left) 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:660: done (1 left) 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:661: done (0 left) 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:662: write (excl) oid 15 current snap is 0 2024-09-13T11:32:29.613 INFO:tasks.rados.rados.0.smithi204.stdout:662: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-13T11:32:29.616 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:32:29.620 INFO:tasks.rados.rados.0.smithi204.stdout:662: writing smithi20438004-15 from 726069 to 1498576 tid 1 2024-09-13T11:32:29.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:32:29.627 INFO:tasks.rados.rados.0.smithi204.stdout:662: writing smithi20438004-15 from 2071117 to 2786793 tid 2 2024-09-13T11:32:29.627 INFO:tasks.rados.rados.0.smithi204.stdout:663: write (excl) oid 26 current snap is 0 2024-09-13T11:32:29.627 INFO:tasks.rados.rados.0.smithi204.stdout:662: finishing write tid 1 to smithi20438004-15 2024-09-13T11:32:29.627 INFO:tasks.rados.rados.0.smithi204.stdout:663: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-13T11:32:29.633 INFO:tasks.rados.rados.0.smithi204.stdout:663: writing smithi20438004-26 from 742876 to 1190632 tid 1 2024-09-13T11:32:29.638 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:29.637+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:29.639 INFO:tasks.rados.rados.0.smithi204.stdout:663: writing smithi20438004-26 from 1729599 to 2305286 tid 2 2024-09-13T11:32:29.645 INFO:tasks.rados.rados.0.smithi204.stdout:663: writing smithi20438004-26 from 3101234 to 3142876 tid 3 2024-09-13T11:32:29.645 INFO:tasks.rados.rados.0.smithi204.stdout:664: read oid 50 snap -1 2024-09-13T11:32:29.645 INFO:tasks.rados.rados.0.smithi204.stdout:664: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: finishing write tid 2 to smithi20438004-15 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:665: read oid 19 snap -1 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:665: expect deleted 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:666: tier_evict oid 5 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:666: tier_evict oid 5 snap -1 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: finishing write tid 3 to smithi20438004-15 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: finishing write tid 4 to smithi20438004-15 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: oid 15 updating version 0 to 28 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: oid 15 updating version 28 to 29 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: oid 15 version 29 is already newer than 27 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 29 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: left oid 15 (ObjNum 67 snap 0 seq_num 67) 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:663: finishing write tid 1 to smithi20438004-26 2024-09-13T11:32:29.646 INFO:tasks.rados.rados.0.smithi204.stdout:662: done (4 left) 2024-09-13T11:32:29.647 INFO:tasks.rados.rados.0.smithi204.stdout:667: read oid 25 snap -1 2024-09-13T11:32:29.647 INFO:tasks.rados.rados.0.smithi204.stdout:667: expect (ObjNum 59 snap 0 seq_num 59) 2024-09-13T11:32:29.647 INFO:tasks.rados.rados.0.smithi204.stdout:668: write (excl) oid 31 current snap is 0 2024-09-13T11:32:29.647 INFO:tasks.rados.rados.0.smithi204.stdout:668: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-13T11:32:29.652 INFO:tasks.rados.rados.0.smithi204.stdout:668: writing smithi20438004-31 from 759683 to 1399041 tid 1 2024-09-13T11:32:29.659 INFO:tasks.rados.rados.0.smithi204.stdout:668: writing smithi20438004-31 from 1904434 to 2623779 tid 2 2024-09-13T11:32:29.662 INFO:tasks.rados.rados.0.smithi204.stdout:668: writing smithi20438004-31 from 3128657 to 3159683 tid 3 2024-09-13T11:32:29.662 INFO:tasks.rados.rados.0.smithi204.stdout:669: write oid 38 current snap is 0 2024-09-13T11:32:29.663 INFO:tasks.rados.rados.0.smithi204.stdout:669: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-13T11:32:29.670 INFO:tasks.rados.rados.0.smithi204.stdout:669: writing smithi20438004-38 from 776490 to 1491097 tid 1 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:669: writing smithi20438004-38 from 2246563 to 2825919 tid 2 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:670: read oid 28 snap -1 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:670: expect deleted 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:671: tier_promote oid 22 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:666: got (22) Invalid argument 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:663: finishing write tid 2 to smithi20438004-26 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:663: finishing write tid 3 to smithi20438004-26 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:663: finishing write tid 4 to smithi20438004-26 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:663: finishing write tid 5 to smithi20438004-26 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:663: oid 26 updating version 0 to 16 2024-09-13T11:32:29.679 INFO:tasks.rados.rados.0.smithi204.stdout:663: oid 26 updating version 16 to 17 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:663: oid 26 version 17 is already newer than 15 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:663: oid 26 version 17 is already newer than 14 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 17 (ObjNum 68 snap 0 seq_num 68) dirty exists 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:663: left oid 26 (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:663: done (8 left) 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:665: done (7 left) 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:666: done (6 left) 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:672: delete oid 41 current snap is 0 2024-09-13T11:32:29.680 INFO:tasks.rados.rados.0.smithi204.stdout:664: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:29.724 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.725 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.733 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:32:29.734 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2293615 2024-09-13T11:32:29.739 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:29.738+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:29.786 INFO:tasks.rados.rados.0.smithi204.stdout:667: expect (ObjNum 59 snap 0 seq_num 59) 2024-09-13T11:32:29.820 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:29.837 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.837 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.840 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:29.839+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:29.846 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:32:29.893 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2617598 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:668: finishing write tid 1 to smithi20438004-31 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:668: finishing write tid 2 to smithi20438004-31 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:664: done (6 left) 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:667: done (5 left) 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:672: done (4 left) 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:673: read oid 14 snap -1 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:673: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:674: write (excl) oid 39 current snap is 0 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:668: finishing write tid 3 to smithi20438004-31 2024-09-13T11:32:29.897 INFO:tasks.rados.rados.0.smithi204.stdout:668: finishing write tid 4 to smithi20438004-31 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:668: finishing write tid 5 to smithi20438004-31 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:668: oid 31 updating version 0 to 19 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:668: oid 31 version 19 is already newer than 17 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:668: oid 31 updating version 19 to 20 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:668: oid 31 version 20 is already newer than 18 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 20 (ObjNum 69 snap 0 seq_num 69) dirty exists 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:668: left oid 31 (ObjNum 69 snap 0 seq_num 69) 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:669: finishing write tid 1 to smithi20438004-38 2024-09-13T11:32:29.898 INFO:tasks.rados.rados.0.smithi204.stdout:674: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-13T11:32:29.902 INFO:tasks.rados.rados.0.smithi204.stdout:674: writing smithi20438004-39 from 410104 to 875209 tid 1 2024-09-13T11:32:29.910 INFO:tasks.rados.rados.0.smithi204.stdout:674: writing smithi20438004-39 from 1563527 to 2146552 tid 2 2024-09-13T11:32:29.917 INFO:tasks.rados.rados.0.smithi204.stdout:674: writing smithi20438004-39 from 2810926 to 3210104 tid 3 2024-09-13T11:32:29.917 INFO:tasks.rados.rados.0.smithi204.stdout:668: done (5 left) 2024-09-13T11:32:29.917 INFO:tasks.rados.rados.0.smithi204.stdout:670: done (4 left) 2024-09-13T11:32:29.917 INFO:tasks.rados.rados.0.smithi204.stdout:675: read oid 34 snap -1 2024-09-13T11:32:29.917 INFO:tasks.rados.rados.0.smithi204.stdout:675: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T11:32:29.917 INFO:tasks.rados.rados.0.smithi204.stdout:676: write (excl) oid 9 current snap is 0 2024-09-13T11:32:29.917 INFO:tasks.rados.rados.0.smithi204.stdout:671: got (2) No such file or directory 2024-09-13T11:32:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:671: got expected ENOENT (src dne) 2024-09-13T11:32:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 29 (ObjNum -513827616 snap 32547 seq_num 304) clean dne 2024-09-13T11:32:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:669: finishing write tid 2 to smithi20438004-38 2024-09-13T11:32:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:676: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-13T11:32:29.923 INFO:tasks.rados.rados.0.smithi204.stdout:676: writing smithi20438004-9 from 426911 to 967265 tid 1 2024-09-13T11:32:29.932 INFO:tasks.rados.rados.0.smithi204.stdout:676: writing smithi20438004-9 from 1622009 to 2348692 tid 2 2024-09-13T11:32:29.937 INFO:tasks.rados.rados.0.smithi204.stdout:676: writing smithi20438004-9 from 3121996 to 3226911 tid 3 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:671: done (5 left) 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:677: read oid 4 snap -1 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:677: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:678: write oid 29 current snap is 0 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:669: finishing write tid 3 to smithi20438004-38 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:669: finishing write tid 4 to smithi20438004-38 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:669: oid 38 updating version 0 to 35 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:669: oid 38 version 35 is already newer than 34 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:669: oid 38 updating version 35 to 36 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 36 (ObjNum 70 snap 0 seq_num 70) dirty exists 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:669: left oid 38 (ObjNum 70 snap 0 seq_num 70) 2024-09-13T11:32:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:678: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-13T11:32:29.942 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:29.941+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:29.944 INFO:tasks.rados.rados.0.smithi204.stdout:678: writing smithi20438004-29 from 443718 to 1059321 tid 1 2024-09-13T11:32:29.950 INFO:tasks.rados.rados.0.smithi204.stdout:678: writing smithi20438004-29 from 1564138 to 2034479 tid 2 2024-09-13T11:32:29.958 INFO:tasks.rados.rados.0.smithi204.stdout:678: writing smithi20438004-29 from 2633066 to 3243718 tid 3 2024-09-13T11:32:29.958 INFO:tasks.rados.rados.0.smithi204.stdout:669: done (6 left) 2024-09-13T11:32:29.958 INFO:tasks.rados.rados.0.smithi204.stdout:679: write oid 37 current snap is 0 2024-09-13T11:32:29.958 INFO:tasks.rados.rados.0.smithi204.stdout:679: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-13T11:32:29.965 INFO:tasks.rados.rados.0.smithi204.stdout:679: writing smithi20438004-37 from 460525 to 1151377 tid 1 2024-09-13T11:32:29.973 INFO:tasks.rados.rados.0.smithi204.stdout:679: writing smithi20438004-37 from 1622620 to 2352972 tid 2 2024-09-13T11:32:29.979 INFO:tasks.rados.rados.0.smithi204.stdout:679: writing smithi20438004-37 from 3060489 to 3260525 tid 3 2024-09-13T11:32:29.979 INFO:tasks.rados.rados.0.smithi204.stdout:680: read oid 11 snap -1 2024-09-13T11:32:29.979 INFO:tasks.rados.rados.0.smithi204.stdout:680: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T11:32:29.979 INFO:tasks.rados.rados.0.smithi204.stdout:681: set_chunk oid 3 target oid 2024-09-13T11:32:29.981 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:29.981 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:681: set_chunk oid 3 offset: 741888 length: 424448 target oid a5d7636f170f977047aefd562ad46649a38a9253d1106a8e65b762ba7d532c8b offset: 0 snap -1 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:682: read oid 27 snap -1 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:682: expect (ObjNum 58 snap 0 seq_num 58) 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:683: read oid 46 snap -1 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:683: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:684: read oid 20 snap -1 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:684: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:29.986 INFO:tasks.rados.rados.0.smithi204.stdout:685: read oid 26 snap -1 2024-09-13T11:32:29.987 INFO:tasks.rados.rados.0.smithi204.stdout:685: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:29.987 INFO:tasks.rados.rados.0.smithi204.stdout:686: write oid 41 current snap is 0 2024-09-13T11:32:29.987 INFO:tasks.rados.rados.0.smithi204.stdout:686: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-13T11:32:29.993 INFO:tasks.rados.rados.0.smithi204.stdout:686: writing smithi20438004-41 from 477332 to 1243433 tid 1 2024-09-13T11:32:29.994 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:32:29.998 INFO:tasks.rados.rados.0.smithi204.stdout:686: writing smithi20438004-41 from 1681102 to 2155112 tid 2 2024-09-13T11:32:30.005 INFO:tasks.rados.rados.0.smithi204.stdout:686: writing smithi20438004-41 from 2687912 to 3277332 tid 3 2024-09-13T11:32:30.005 INFO:tasks.rados.rados.0.smithi204.stdout:687: write (excl) oid 19 current snap is 0 2024-09-13T11:32:30.005 INFO:tasks.rados.rados.0.smithi204.stdout:673: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-13T11:32:30.043 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:30.042+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:30.070 INFO:tasks.rados.rados.0.smithi204.stdout:674: finishing write tid 1 to smithi20438004-39 2024-09-13T11:32:30.070 INFO:tasks.rados.rados.0.smithi204.stdout:674: finishing write tid 2 to smithi20438004-39 2024-09-13T11:32:30.070 INFO:tasks.rados.rados.0.smithi204.stdout:687: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-13T11:32:30.075 INFO:tasks.rados.rados.0.smithi204.stdout:687: writing smithi20438004-19 from 494139 to 1051842 tid 1 2024-09-13T11:32:30.081 INFO:tasks.rados.rados.0.smithi204.stdout:687: writing smithi20438004-19 from 1455937 to 2189958 tid 2 2024-09-13T11:32:30.088 INFO:tasks.rados.rados.0.smithi204.stdout:687: writing smithi20438004-19 from 2948041 to 3294139 tid 3 2024-09-13T11:32:30.088 INFO:tasks.rados.rados.0.smithi204.stdout:673: done (14 left) 2024-09-13T11:32:30.088 INFO:tasks.rados.rados.0.smithi204.stdout:688: write (excl) oid 40 current snap is 0 2024-09-13T11:32:30.088 INFO:tasks.rados.rados.0.smithi204.stdout:688: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-09-13T11:32:30.094 INFO:tasks.rados.rados.0.smithi204.stdout:688: writing smithi20438004-40 from 510946 to 1143898 tid 1 2024-09-13T11:32:30.101 INFO:tasks.rados.rados.0.smithi204.stdout:688: writing smithi20438004-40 from 1798066 to 2275745 tid 2 2024-09-13T11:32:30.102 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:30.102 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:30.105 INFO:tasks.rados.rados.0.smithi204.stdout:688: writing smithi20438004-40 from 2742758 to 3199975 tid 3 2024-09-13T11:32:30.105 INFO:tasks.rados.rados.0.smithi204.stdout:689: write (excl) oid 48 current snap is 0 2024-09-13T11:32:30.105 INFO:tasks.rados.rados.0.smithi204.stdout:674: finishing write tid 3 to smithi20438004-39 2024-09-13T11:32:30.105 INFO:tasks.rados.rados.0.smithi204.stdout:674: finishing write tid 4 to smithi20438004-39 2024-09-13T11:32:30.105 INFO:tasks.rados.rados.0.smithi204.stdout:689: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-13T11:32:30.111 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:32:30.112 INFO:tasks.rados.rados.0.smithi204.stdout:689: writing smithi20438004-48 from 527753 to 1235954 tid 1 2024-09-13T11:32:30.119 INFO:tasks.rados.rados.0.smithi204.stdout:689: writing smithi20438004-48 from 1856548 to 2594238 tid 2 2024-09-13T11:32:30.123 INFO:tasks.rados.rados.0.smithi204.stdout:689: writing smithi20438004-48 from 3286534 to 3327753 tid 3 2024-09-13T11:32:30.123 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:674: finishing write tid 5 to smithi20438004-39 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:674: oid 39 updating version 0 to 35 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:674: oid 39 version 35 is already newer than 34 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:674: oid 39 updating version 35 to 37 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:674: oid 39 version 37 is already newer than 36 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 37 (ObjNum 72 snap 0 seq_num 72) dirty exists 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:674: left oid 39 (ObjNum 72 snap 0 seq_num 72) 2024-09-13T11:32:30.124 INFO:tasks.rados.rados.0.smithi204.stdout:675: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T11:32:30.144 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:30.143+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:30.181 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1414939 2024-09-13T11:32:30.199 INFO:tasks.rados.rados.0.smithi204.stdout:676: finishing write tid 1 to smithi20438004-9 2024-09-13T11:32:30.199 INFO:tasks.rados.rados.0.smithi204.stdout:674: done (15 left) 2024-09-13T11:32:30.199 INFO:tasks.rados.rados.0.smithi204.stdout:675: done (14 left) 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:690: delete oid 30 current snap is 0 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: finishing write tid 2 to smithi20438004-9 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: finishing write tid 3 to smithi20438004-9 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: finishing write tid 4 to smithi20438004-9 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: finishing write tid 5 to smithi20438004-9 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: oid 9 updating version 0 to 23 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: oid 9 version 23 is already newer than 22 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: oid 9 updating version 23 to 24 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: oid 9 version 24 is already newer than 21 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 24 (ObjNum 73 snap 0 seq_num 73) dirty exists 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:676: left oid 9 (ObjNum 73 snap 0 seq_num 73) 2024-09-13T11:32:30.200 INFO:tasks.rados.rados.0.smithi204.stdout:677: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:30.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:30.217 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:30.217 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:30.222 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:30.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:32:30.245 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:32:30.244+0000 7f0a80816640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32882) UID: 0 2024-09-13T11:32:30.251 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 897588 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:678: finishing write tid 1 to smithi20438004-29 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:676: done (14 left) 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:677: done (13 left) 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:690: done (12 left) 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:691: read oid 22 snap -1 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:691: expect deleted 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:692: tier_evict oid 10 2024-09-13T11:32:30.265 INFO:tasks.rados.rados.0.smithi204.stdout:678: finishing write tid 2 to smithi20438004-29 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: finishing write tid 3 to smithi20438004-29 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: finishing write tid 4 to smithi20438004-29 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: finishing write tid 5 to smithi20438004-29 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: oid 29 updating version 0 to 38 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: oid 29 version 38 is already newer than 37 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: oid 29 updating version 38 to 39 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: oid 29 updating version 39 to 40 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 40 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: left oid 29 (ObjNum 74 snap 0 seq_num 74) 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:692: tier_evict oid 10 snap -1 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:679: finishing write tid 1 to smithi20438004-37 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:679: finishing write tid 2 to smithi20438004-37 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:678: done (13 left) 2024-09-13T11:32:30.266 INFO:tasks.rados.rados.0.smithi204.stdout:693: read oid 28 snap -1 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:693: expect deleted 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:694: write (excl) oid 18 current snap is 0 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: finishing write tid 3 to smithi20438004-37 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: finishing write tid 4 to smithi20438004-37 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: finishing write tid 5 to smithi20438004-37 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: oid 37 updating version 0 to 17 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: oid 37 updating version 17 to 18 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: oid 37 updating version 18 to 19 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: oid 37 version 19 is already newer than 16 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 19 (ObjNum 75 snap 0 seq_num 75) dirty exists 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:679: left oid 37 (ObjNum 75 snap 0 seq_num 75) 2024-09-13T11:32:30.267 INFO:tasks.rados.rados.0.smithi204.stdout:694: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-13T11:32:30.269 INFO:tasks.rados.rados.0.smithi204.stdout:694: writing smithi20438004-18 from 561367 to 1020066 tid 1 2024-09-13T11:32:30.277 INFO:tasks.rados.rados.0.smithi204.stdout:694: writing smithi20438004-18 from 1573512 to 2314871 tid 2 2024-09-13T11:32:30.283 INFO:tasks.rados.rados.0.smithi204.stdout:694: writing smithi20438004-18 from 2941380 to 3361367 tid 3 2024-09-13T11:32:30.283 INFO:tasks.rados.rados.0.smithi204.stdout:679: done (14 left) 2024-09-13T11:32:30.283 INFO:tasks.rados.rados.0.smithi204.stdout:695: tier_promote oid 25 2024-09-13T11:32:30.283 INFO:tasks.rados.rados.0.smithi204.stdout:681: finishing set_chunk to oid 3 2024-09-13T11:32:30.283 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 741888 length: 424448 2024-09-13T11:32:30.283 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-13T11:32:30.283 INFO:tasks.rados.rados.0.smithi204.stdout:680: expect (ObjNum 10 snap 0 seq_num 10) 2024-09-13T11:32:30.284 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:30.332 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:30.334 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:30.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:30.346 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:30.345+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:30.352 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:30.357 INFO:tasks.rados.rados.0.smithi204.stdout:684: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:30.357 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:30.357 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:30.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:32:30.369 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-09-13T11:32:30.369 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-13T11:32:30.385 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 305130 2024-09-13T11:32:30.386 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:30.386 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:30.403 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:30.403 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:30.420 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:30.420 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:30.435 INFO:tasks.rados.rados.0.smithi204.stdout:683: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T11:32:30.437 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:30.437 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:30.443 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1413691 2024-09-13T11:32:30.446 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:30.445+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:30.454 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:30.454 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:30.457 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:30.457 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:32:30.457 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:32:30.457 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:30.458 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T11:32:30.458 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:32:30.458 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:30.471 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:30.471 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:30.475 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:32:30.476 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:30.476 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:30.486 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T11:32:30.486 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-09-13T11:32:30.486 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:32:30.486 DEBUG:teuthology.orchestra.run.smithi154:> type ceph-objectstore-tool 2024-09-13T11:32:30.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:32:30.492 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:30.492 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:30.501 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2639812 2024-09-13T11:32:30.502 INFO:tasks.rados.rados.0.smithi204.stdout:685: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:30.509 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:30.509 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:30.526 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:30.526 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:30.541 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:32:30.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-09-13T11:32:30.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:30.542 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 142301 2024-09-13T11:32:30.559 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:30.559 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:30.576 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:30.576 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:30.577 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:30.576+0000 7fe596463640 0 -- 172.21.15.38:0/571433868 >> v2:172.21.15.204:6812/501019151 conn(0x7fe590220230 msgr2=0x7fe590222620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:30.582 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:30.581+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:30.593 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:30.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:30.600 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:30.600 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:30.600 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:30.600 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:30.600 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:30.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:32:30.632 INFO:tasks.rados.rados.0.smithi204.stdout:682: expect (ObjNum 58 snap 0 seq_num 58) 2024-09-13T11:32:30.648 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:30.647+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:30.710 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 221565 2024-09-13T11:32:30.749 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:30.748+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:30.754 INFO:tasks.rados.rados.0.smithi204.stdout:686: finishing write tid 1 to smithi20438004-41 2024-09-13T11:32:30.754 INFO:tasks.rados.rados.0.smithi204.stdout:686: finishing write tid 2 to smithi20438004-41 2024-09-13T11:32:30.754 INFO:tasks.rados.rados.0.smithi204.stdout:686: finishing write tid 3 to smithi20438004-41 2024-09-13T11:32:30.754 INFO:tasks.rados.rados.0.smithi204.stdout:686: finishing write tid 4 to smithi20438004-41 2024-09-13T11:32:30.754 INFO:tasks.rados.rados.0.smithi204.stdout:686: finishing write tid 5 to smithi20438004-41 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:686: oid 41 updating version 0 to 37 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:686: oid 41 updating version 37 to 39 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:686: oid 41 version 39 is already newer than 38 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:686: oid 41 version 39 is already newer than 36 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 39 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:686: left oid 41 (ObjNum 76 snap 0 seq_num 76) 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:687: finishing write tid 1 to smithi20438004-19 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:680: done (14 left) 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:681: done (13 left) 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:682: done (12 left) 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:683: done (11 left) 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:684: done (10 left) 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:685: done (9 left) 2024-09-13T11:32:30.755 INFO:tasks.rados.rados.0.smithi204.stdout:686: done (8 left) 2024-09-13T11:32:30.756 INFO:tasks.rados.rados.0.smithi204.stdout:696: read oid 26 snap -1 2024-09-13T11:32:30.756 INFO:tasks.rados.rados.0.smithi204.stdout:696: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:30.756 INFO:tasks.rados.rados.0.smithi204.stdout:697: read oid 5 snap -1 2024-09-13T11:32:30.756 INFO:tasks.rados.rados.0.smithi204.stdout:697: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-13T11:32:30.756 INFO:tasks.rados.rados.0.smithi204.stdout:698: write (excl) oid 45 current snap is 0 2024-09-13T11:32:30.756 INFO:tasks.rados.rados.0.smithi204.stdout:687: finishing write tid 2 to smithi20438004-19 2024-09-13T11:32:30.756 INFO:tasks.rados.rados.0.smithi204.stdout:698: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-13T11:32:30.759 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:30.759 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'23)", 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.977654364999999, 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005604375, 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779709+0000", 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.63e-07 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779721+0000", 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.204e-05 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779728+0000", 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4150000000000002e-06 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779732+0000", 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6839999999999998e-06 2024-09-13T11:32:30.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779761+0000", 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9405999999999999e-05 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779773+0000", 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1398e-05 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785264+0000", 2024-09-13T11:32:30.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054913089999999998 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785313+0000", 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8659999999999998e-05 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:44 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'26, mlcod=24'23)", 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.977084278, 2024-09-13T11:32:30.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055622880000000003, 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780279+0000", 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780365+0000", 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6462e-05 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780372+0000", 2024-09-13T11:32:30.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780376+0000", 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784152+0000", 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037767650000000001 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784173+0000", 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0899000000000001e-05 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785814+0000", 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001640561 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785841+0000", 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6628e-05 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:45 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'27, mlcod=24'23)", 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.976496840999999, 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005510928, 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4999999999999998e-07 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780870+0000", 2024-09-13T11:32:30.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6940000000000001e-06 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780877+0000", 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0249999999999997e-06 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780883+0000", 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784553+0000", 2024-09-13T11:32:30.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036697689999999998 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784573+0000", 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9936999999999999e-05 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786355+0000", 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017819870000000001 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786377+0000", 2024-09-13T11:32:30.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1863000000000001e-05 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:46 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'28, mlcod=24'23)", 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.975901579999999, 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049328109999999996, 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7800000000000002e-07 2024-09-13T11:32:30.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781471+0000", 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.533e-06 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781477+0000", 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2090000000000002e-06 2024-09-13T11:32:30.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781483+0000", 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4770000000000002e-06 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784808+0000", 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033250839999999999 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784826+0000", 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8161000000000001e-05 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786379+0000", 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001552905 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786394+0000", 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5064e-05 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:91 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'65, mlcod=24'61)", 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.775052577, 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053956129999999996, 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3700000000000001e-07 2024-09-13T11:32:30.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982314+0000", 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2229999999999999e-06 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982320+0000", 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6499999999999999e-06 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982324+0000", 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6780000000000002e-06 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.984345+0000", 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020208819999999999 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.984364+0000", 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8834e-05 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.987671+0000", 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033072689999999998 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.987706+0000", 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.464e-05 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:30.782 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.020182866999999, 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:30.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'105, mlcod=24'100)", 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:30.787 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.019619263999999, 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061709499999999997, 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:30.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737744+0000", 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7700000000000004e-07 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737749+0000", 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737756+0000", 2024-09-13T11:32:30.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7649999999999996e-06 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737760+0000", 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6650000000000001e-06 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742613+0000", 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048528909999999998 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742629+0000", 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6481e-05 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743902+0000", 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012723630000000001 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743914+0000", 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2663999999999999e-05 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:140 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'106, mlcod=25'101)", 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.018982880999999, 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005547064, 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738382+0000", 2024-09-13T11:32:30.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5909999999999998e-06 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738386+0000", 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5240000000000001e-06 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738392+0000", 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8660000000000002e-06 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738396+0000", 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6739999999999996e-06 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742884+0000", 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004487441 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742896+0000", 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1866e-05 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743915+0000", 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010193229999999999 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743927+0000", 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1779e-05 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:141 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'107, mlcod=25'101)", 2024-09-13T11:32:30.796 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.739053+0000", 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.018309079000002, 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050398209999999999, 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739053+0000", 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739053+0000", 2024-09-13T11:32:30.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739055+0000", 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6270000000000001e-06 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739104+0000", 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8699000000000002e-05 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739128+0000", 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3975999999999999e-05 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739140+0000", 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2027e-05 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743177+0000", 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004037311 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743187+0000", 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8660000000000001e-06 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744070+0000", 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00088299899999999998 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744093+0000", 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3315999999999998e-05 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:142 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'108, mlcod=25'101)", 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.739436+0000", 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.017926117999998, 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049474050000000002, 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739436+0000", 2024-09-13T11:32:30.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739436+0000", 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739439+0000", 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0110000000000001e-06 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739443+0000", 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6389999999999998e-06 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739449+0000", 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4010000000000002e-06 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739453+0000", 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3979999999999999e-06 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743719+0000", 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042658669999999996 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743738+0000", 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9113000000000001e-05 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744357+0000", 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061914000000000001 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744384+0000", 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6835999999999999e-05 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.805 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.2802572400000001, 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:32:30.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:32:30.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:30.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.812 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.2751986820000001, 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.27477486, 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.819 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:32:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:32:30.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.274756816, 2024-09-13T11:32:30.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:32:30.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:32:30.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:32:30.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:32 4.9 4:9441befd:::smithi20438004-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.540583+0000", 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.21677926, 2024-09-13T11:32:30.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049216040000000004, 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 32 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.540583+0000", 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.540583+0000", 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.540584+0000", 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1899999999999998e-07 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.540585+0000", 2024-09-13T11:32:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.339e-06 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.540590+0000", 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5929999999999999e-06 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.540593+0000", 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3069999999999998e-06 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.541869+0000", 2024-09-13T11:32:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001276382 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.541915+0000", 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5389999999999997e-05 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.545299+0000", 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003384509 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.545383+0000", 2024-09-13T11:32:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3659000000000007e-05 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.545505+0000", 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000122106 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.13.23:1 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.085933+0000", 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6714290590000003, 2024-09-13T11:32:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050668559999999998, 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085933+0000", 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085933+0000", 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085934+0000", 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7199999999999997e-07 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085939+0000", 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7899999999999999e-06 2024-09-13T11:32:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085948+0000", 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9020000000000005e-06 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085956+0000", 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.807e-06 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.088108+0000", 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021517509999999999 2024-09-13T11:32:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.088241+0000", 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013326000000000001 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.091000+0000", 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002759474 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5585160440000001, 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:32:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:32:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:32:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.544675255, 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:32:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:402 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.274893+0000", 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4824699849999998, 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053214830000000001, 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 402 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274893+0000", 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274893+0000", 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274894+0000", 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274899+0000", 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1599999999999997e-06 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274913+0000", 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4051e-05 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274921+0000", 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2709999999999992e-06 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274980+0000", 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9045999999999997e-05 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.275062+0000", 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1589999999999996e-05 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.277565+0000", 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002503857 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.278344+0000", 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077808900000000004 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.278489+0000", 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014568599999999999 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.280097+0000", 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001607598 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.280155+0000", 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8400000000000003e-05 2024-09-13T11:32:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.280214+0000", 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8739000000000001e-05 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4457072809999998, 2024-09-13T11:32:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:32:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:30.864 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:30.866 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:30.864+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:30.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:32:30.872 INFO:tasks.rados.rados.0.smithi204.stdout:698: writing smithi20438004-45 from 578174 to 1112122 tid 1 2024-09-13T11:32:30.872 INFO:tasks.rados.rados.0.smithi204.stdout:698: writing smithi20438004-45 from 1515641 to 2000658 tid 2 2024-09-13T11:32:30.872 INFO:tasks.rados.rados.0.smithi204.stdout:698: writing smithi20438004-45 from 2736097 to 3238402 tid 3 2024-09-13T11:32:30.872 INFO:tasks.rados.rados.0.smithi204.stdout:699: write oid 39 current snap is 0 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: finishing write tid 3 to smithi20438004-19 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: finishing write tid 4 to smithi20438004-19 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: finishing write tid 5 to smithi20438004-19 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: oid 19 updating version 0 to 39 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: oid 19 updating version 39 to 40 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: oid 19 updating version 40 to 41 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: oid 19 version 41 is already newer than 38 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 41 (ObjNum 77 snap 0 seq_num 77) dirty exists 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:687: left oid 19 (ObjNum 77 snap 0 seq_num 77) 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:688: finishing write tid 1 to smithi20438004-40 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:688: finishing write tid 2 to smithi20438004-40 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:699: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:699: writing smithi20438004-39 from 594981 to 1204178 tid 1 2024-09-13T11:32:30.873 INFO:tasks.rados.rados.0.smithi204.stdout:699: writing smithi20438004-39 from 1974123 to 2719151 tid 2 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:699: writing smithi20438004-39 from 3279873 to 3394981 tid 3 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:687: done (11 left) 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:700: set_chunk oid 41 target oid 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:700: set_chunk oid 41 offset: 1816064 length: 343552 target oid 64495a2c08186d99c7257ed54ab8d7b6a2f5779ed98e39a7912122e90f62c579 offset: 0 snap -1 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:701: write (excl) oid 43 current snap is 0 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:701: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:701: writing smithi20438004-43 from 611788 to 1012587 tid 1 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:701: writing smithi20438004-43 from 1748958 to 2237644 tid 2 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:701: writing smithi20438004-43 from 2907296 to 3411788 tid 3 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:702: write (excl) oid 23 current snap is 0 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:688: finishing write tid 3 to smithi20438004-40 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:688: finishing write tid 4 to smithi20438004-40 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:702: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:702: writing smithi20438004-23 from 628595 to 1104643 tid 1 2024-09-13T11:32:30.874 INFO:tasks.rados.rados.0.smithi204.stdout:702: writing smithi20438004-23 from 1807440 to 2439784 tid 2 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:702: writing smithi20438004-23 from 2934719 to 3387193 tid 3 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:703: read oid 42 snap -1 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:703: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:704: write (excl) oid 12 current snap is 0 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:688: finishing write tid 5 to smithi20438004-40 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:688: oid 40 updating version 0 to 34 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:688: oid 40 version 34 is already newer than 31 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:688: oid 40 version 34 is already newer than 33 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:688: oid 40 version 34 is already newer than 32 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 34 (ObjNum 78 snap 0 seq_num 78) dirty exists 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:688: left oid 40 (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:689: finishing write tid 1 to smithi20438004-48 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:704: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-13T11:32:30.875 INFO:tasks.rados.rados.0.smithi204.stdout:704: writing smithi20438004-12 from 645402 to 1196699 tid 1 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:704: writing smithi20438004-12 from 1749569 to 2241924 tid 2 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:704: writing smithi20438004-12 from 2845789 to 3445402 tid 3 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:688: done (15 left) 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:705: read oid 44 snap -1 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:705: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: finishing write tid 2 to smithi20438004-48 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: finishing write tid 3 to smithi20438004-48 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: finishing write tid 4 to smithi20438004-48 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: finishing write tid 5 to smithi20438004-48 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: oid 48 updating version 0 to 25 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: oid 48 version 25 is already newer than 24 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: oid 48 updating version 25 to 27 2024-09-13T11:32:30.876 INFO:tasks.rados.rados.0.smithi204.stdout:689: oid 48 version 27 is already newer than 26 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 27 (ObjNum 79 snap 0 seq_num 79) dirty exists 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:689: left oid 48 (ObjNum 79 snap 0 seq_num 79) 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:692: got (22) Invalid argument 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:694: finishing write tid 1 to smithi20438004-18 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:694: finishing write tid 2 to smithi20438004-18 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:694: finishing write tid 3 to smithi20438004-18 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:689: done (15 left) 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:691: done (14 left) 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:692: done (13 left) 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:693: done (12 left) 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:706: read oid 31 snap -1 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:706: expect (ObjNum 69 snap 0 seq_num 69) 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:707: read oid 47 snap -1 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:707: expect (ObjNum 61 snap 0 seq_num 61) 2024-09-13T11:32:30.877 INFO:tasks.rados.rados.0.smithi204.stdout:708: read oid 46 snap -1 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:708: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:709: read oid 40 snap -1 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:709: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:695: got (0) Success 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 33 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:694: finishing write tid 4 to smithi20438004-18 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:694: finishing write tid 5 to smithi20438004-18 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:694: oid 18 updating version 0 to 43 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:694: oid 18 version 43 is already newer than 40 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:694: oid 18 version 43 is already newer than 42 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:694: oid 18 version 43 is already newer than 41 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 43 (ObjNum 81 snap 0 seq_num 81) dirty exists 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:694: left oid 18 (ObjNum 81 snap 0 seq_num 81) 2024-09-13T11:32:30.878 INFO:tasks.rados.rados.0.smithi204.stdout:696: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:30.879 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1789232 2024-09-13T11:32:30.906 INFO:teuthology.orchestra.run.smithi154.stdout:4.4 2024-09-13T11:32:30.906 INFO:teuthology.orchestra.run.smithi154.stdout:3.3 2024-09-13T11:32:30.906 INFO:teuthology.orchestra.run.smithi154.stdout:2.5 2024-09-13T11:32:30.942 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:30.966 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:30.965+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:30.977 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:30.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:30.978 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:30.978 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:30.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:32:30.989 INFO:tasks.rados.rados.0.smithi204.stdout:697: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-13T11:32:31.010 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2310959 2024-09-13T11:32:31.023 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:31.067 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:31.066+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:31.070 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:31.094 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:31.094 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:31.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:32:31.118 INFO:tasks.rados.rados.0.smithi204.stdout:698: finishing write tid 1 to smithi20438004-45 2024-09-13T11:32:31.118 INFO:tasks.rados.rados.0.smithi204.stdout:698: finishing write tid 2 to smithi20438004-45 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: finishing write tid 3 to smithi20438004-45 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: finishing write tid 4 to smithi20438004-45 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: finishing write tid 5 to smithi20438004-45 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: oid 45 updating version 0 to 23 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: oid 45 updating version 23 to 24 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: oid 45 version 24 is already newer than 22 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: oid 45 version 24 is already newer than 21 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 24 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:698: left oid 45 (ObjNum 82 snap 0 seq_num 82) 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:699: finishing write tid 1 to smithi20438004-39 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:694: done (15 left) 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:695: done (14 left) 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:696: done (13 left) 2024-09-13T11:32:31.119 INFO:tasks.rados.rados.0.smithi204.stdout:697: done (12 left) 2024-09-13T11:32:31.120 INFO:tasks.rados.rados.0.smithi204.stdout:698: done (11 left) 2024-09-13T11:32:31.120 INFO:tasks.rados.rados.0.smithi204.stdout:710: read oid 16 snap -1 2024-09-13T11:32:31.120 INFO:tasks.rados.rados.0.smithi204.stdout:710: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-13T11:32:31.120 INFO:tasks.rados.rados.0.smithi204.stdout:711: set_chunk oid 49 target oid 2024-09-13T11:32:31.124 INFO:tasks.rados.rados.0.smithi204.stdout:711: set_chunk oid 49 offset: 770560 length: 270336 target oid 80b2bf02fda5674fe681b80152e22b86c98440432ea21889a7e82e125193b9e5 offset: 0 snap -1 2024-09-13T11:32:31.125 INFO:tasks.rados.rados.0.smithi204.stdout:712: read oid 4 snap -1 2024-09-13T11:32:31.125 INFO:tasks.rados.rados.0.smithi204.stdout:712: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:31.125 INFO:tasks.rados.rados.0.smithi204.stdout:713: read oid 28 snap -1 2024-09-13T11:32:31.125 INFO:tasks.rados.rados.0.smithi204.stdout:713: expect deleted 2024-09-13T11:32:31.125 INFO:tasks.rados.rados.0.smithi204.stdout:714: set_chunk oid 8 target oid 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:714: set_chunk oid 8 offset: 500224 length: 1157120 target oid ed62dd31e2f024c7f787b307b94c57f47e3ecdb68d4af7d43728e824e16351b4 offset: 0 snap -1 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:699: finishing write tid 2 to smithi20438004-39 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:699: finishing write tid 3 to smithi20438004-39 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:699: finishing write tid 4 to smithi20438004-39 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:699: finishing write tid 5 to smithi20438004-39 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:699: oid 39 updating version 0 to 45 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:699: oid 39 version 45 is already newer than 44 2024-09-13T11:32:31.137 INFO:tasks.rados.rados.0.smithi204.stdout:699: oid 39 updating version 45 to 46 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:699: oid 39 updating version 46 to 47 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 47 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:699: left oid 39 (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:700: finishing set_chunk to oid 41 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 39 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:701: finishing write tid 1 to smithi20438004-43 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:701: finishing write tid 2 to smithi20438004-43 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:701: finishing write tid 3 to smithi20438004-43 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:699: done (15 left) 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:700: done (14 left) 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:715: write (excl) oid 30 current snap is 0 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:701: finishing write tid 4 to smithi20438004-43 2024-09-13T11:32:31.138 INFO:tasks.rados.rados.0.smithi204.stdout:715: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-13T11:32:31.143 INFO:tasks.rados.rados.0.smithi204.stdout:715: writing smithi20438004-30 from 662209 to 1288755 tid 1 2024-09-13T11:32:31.149 INFO:tasks.rados.rados.0.smithi204.stdout:715: writing smithi20438004-30 from 1808051 to 2444064 tid 2 2024-09-13T11:32:31.154 INFO:tasks.rados.rados.0.smithi204.stdout:715: writing smithi20438004-30 from 2873212 to 3462209 tid 3 2024-09-13T11:32:31.154 INFO:tasks.rados.rados.0.smithi204.stdout:716: set_chunk oid 24 target oid 2024-09-13T11:32:31.169 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:31.167+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:31.174 INFO:tasks.rados.rados.0.smithi204.stdout:716: set_chunk oid 24 offset: 84480 length: 2301952 target oid 68ec8fc88385ac82d10ab7ba1a92c088a080b24818a473ec0906443c4d32b32d offset: 0 snap -1 2024-09-13T11:32:31.174 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:701: finishing write tid 5 to smithi20438004-43 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:701: oid 43 updating version 0 to 45 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:701: oid 43 updating version 45 to 46 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:701: oid 43 updating version 46 to 48 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:701: oid 43 version 48 is already newer than 47 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 48 (ObjNum 84 snap 0 seq_num 84) dirty exists 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:701: left oid 43 (ObjNum 84 snap 0 seq_num 84) 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:702: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:702: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:31.175 INFO:tasks.rados.rados.0.smithi204.stdout:703: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T11:32:31.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 4.4 --file /var/log/ceph/exp.4.4.2 2024-09-13T11:32:31.220 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:31.220 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:31.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:31.220 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:31.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6950424870000003, 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.221 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:31.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:32:31.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:32:31.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:26 4.8 4:1cf41451:::smithi20438004-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6895110750000004, 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083581760000000001, 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 26 2024-09-13T11:32:31.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528196+0000", 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.048e-06 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528601+0000", 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040467299999999998 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528612+0000", 2024-09-13T11:32:31.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.171e-05 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528620+0000", 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1140000000000006e-06 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532030+0000", 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034098779999999999 2024-09-13T11:32:31.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532128+0000", 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7910000000000003e-05 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532280+0000", 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015168700000000001 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533907+0000", 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016271 2024-09-13T11:32:31.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533914+0000", 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2590000000000003e-06 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536475+0000", 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025609259999999998 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536517+0000", 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.163e-05 2024-09-13T11:32:31.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536553+0000", 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6241000000000001e-05 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:27 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6890739430000004, 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082102310000000001, 2024-09-13T11:32:31.235 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 27 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.51e-07 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528635+0000", 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8399999999999999e-06 2024-09-13T11:32:31.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528639+0000", 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528642+0000", 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.19e-06 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533286+0000", 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046437859999999996 2024-09-13T11:32:31.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533371+0000", 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4900000000000004e-05 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533475+0000", 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104285 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534168+0000", 2024-09-13T11:32:31.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069284800000000003 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534400+0000", 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023182699999999999 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536780+0000", 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023795880000000002 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536809+0000", 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9156e-05 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536842+0000", 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3546e-05 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:28 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:31.241 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6890489740000003, 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083368629999999999, 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 28 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528659+0000", 2024-09-13T11:32:31.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528664+0000", 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.42e-06 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528675+0000", 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1447999999999999e-05 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533704+0000", 2024-09-13T11:32:31.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050284769999999999 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533777+0000", 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3675000000000002e-05 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536883+0000", 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031062820000000001 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536929+0000", 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5237000000000001e-05 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536994+0000", 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5152000000000002e-05 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:31.246 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0061043459999999, 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:32:31.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:32:31.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.252 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9977621480000001, 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:32:31.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:32:31.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9774041449999999, 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:32:31.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:32:31.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9696033860000002, 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:32:31.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:32:31.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:32:31.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:394 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.274764+0000", 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9429419569999999, 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068927859999999997, 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.271 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 394 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274764+0000", 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274764+0000", 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274765+0000", 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.92e-07 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274769+0000", 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1309999999999999e-06 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274779+0000", 2024-09-13T11:32:31.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7170000000000003e-06 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274784+0000", 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3319999999999999e-06 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274831+0000", 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6808999999999999e-05 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274890+0000", 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9095000000000003e-05 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.277414+0000", 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025245889999999998 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.278135+0000", 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072055700000000003 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.278665+0000", 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052985100000000002 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.281588+0000", 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029231700000000001 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.281619+0000", 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1390999999999998e-05 2024-09-13T11:32:31.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.281657+0000", 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7351999999999997e-05 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:456 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.287632+0000", 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9300741879999999, 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072535869999999997, 2024-09-13T11:32:31.277 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 456 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287632+0000", 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287632+0000", 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287633+0000", 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000002e-07 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287637+0000", 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2830000000000003e-06 2024-09-13T11:32:31.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287647+0000", 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0040999999999999e-05 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287652+0000", 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.31e-06 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287706+0000", 2024-09-13T11:32:31.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4103e-05 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.287770+0000", 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4129999999999998e-05 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.290044+0000", 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022737930000000001 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.290728+0000", 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068422100000000005 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.290870+0000", 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014162499999999999 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.294816+0000", 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039462719999999998 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.294846+0000", 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0134999999999999e-05 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.294885+0000", 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8899999999999997e-05 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:494 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.289158+0000", 2024-09-13T11:32:31.283 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9285484249999998, 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061373599999999997, 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 494 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289158+0000", 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289158+0000", 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289158+0000", 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1999999999999999e-07 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289162+0000", 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0600000000000001e-06 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289172+0000", 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0054999999999999e-05 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289177+0000", 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9649999999999999e-06 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289224+0000", 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6798000000000002e-05 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.289288+0000", 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3857999999999997e-05 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.292297+0000", 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030089449999999998 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.292764+0000", 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000467071 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.293041+0000", 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027712499999999997 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.295235+0000", 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002194002 2024-09-13T11:32:31.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.295261+0000", 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6211999999999999e-05 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.295295+0000", 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3549e-05 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4200965779999999, 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:32:31.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:32:31.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:796 4.8 4:1cf41451:::smithi20438004-7:head [write 1826593~546335] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.807752+0000", 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.409954216, 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007227374, 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.296 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 796 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807752+0000", 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807752+0000", 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807753+0000", 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3099999999999996e-07 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810912+0000", 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031597629999999999 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810932+0000", 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9272000000000001e-05 2024-09-13T11:32:31.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810943+0000", 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1474999999999999e-05 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811018+0000", 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4581999999999998e-05 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811109+0000", 2024-09-13T11:32:31.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0989000000000006e-05 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811231+0000", 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012212599999999999 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814605+0000", 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033741999999999999 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814676+0000", 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0789000000000003e-05 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814815+0000", 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013907599999999999 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814929+0000", 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114064 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814979+0000", 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0207000000000002e-05 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:797 4.8 4:1cf41451:::smithi20438004-7:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4067340150000001, 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086937119999999993, 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.302 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 797 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:31.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810974+0000", 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7010000000000001e-06 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811111+0000", 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013712699999999999 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811123+0000", 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1854e-05 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811131+0000", 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3979999999999999e-06 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813131+0000", 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019997740000000002 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813239+0000", 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108378 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813421+0000", 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018227099999999999 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814991+0000", 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015690599999999999 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816465+0000", 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001474784 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819575+0000", 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031099769999999999 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819626+0000", 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0609999999999998e-05 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819666+0000", 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9777999999999998e-05 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:798 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.406560389, 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087603679999999993, 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 798 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811150+0000", 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811157+0000", 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811163+0000", 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813998+0000", 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002835382 2024-09-13T11:32:31.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814069+0000", 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1099000000000005e-05 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814212+0000", 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142647 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816475+0000", 2024-09-13T11:32:31.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022629799999999999 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816520+0000", 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5099999999999998e-05 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819739+0000", 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032186570000000002 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819820+0000", 2024-09-13T11:32:31.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1527999999999996e-05 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819906+0000", 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5845000000000002e-05 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:799 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4065190489999999, 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089993910000000007, 2024-09-13T11:32:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 799 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811188+0000", 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811192+0000", 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.566e-06 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811202+0000", 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4790000000000002e-06 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811209+0000", 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814715+0000", 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035061150000000002 2024-09-13T11:32:31.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814798+0000", 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2854000000000003e-05 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819962+0000", 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051643419999999997 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820067+0000", 2024-09-13T11:32:31.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010525 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820186+0000", 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118886 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:804 4.8 4:1d367c64:::smithi20438004-45:head [read 0~2739508,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811243+0000", 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4064628369999999, 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068358810000000002, 2024-09-13T11:32:31.319 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 804 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811243+0000", 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811243+0000", 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811244+0000", 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9099999999999999e-07 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811246+0000", 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4399999999999999e-06 2024-09-13T11:32:31.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811256+0000", 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0045e-05 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811260+0000", 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7230000000000002e-06 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816554+0000", 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052941680000000001 2024-09-13T11:32:31.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816620+0000", 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5988000000000006e-05 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818079+0000", 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001459126 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:805 4.8 4:1d367c64:::smithi20438004-45:head [read 0~2739508,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4064417090000001, 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082479100000000007, 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 805 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811265+0000", 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5400000000000002e-07 2024-09-13T11:32:31.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811266+0000", 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.939e-06 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811270+0000", 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4319999999999999e-06 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811274+0000", 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4150000000000001e-06 2024-09-13T11:32:31.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818160+0000", 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068854700000000003 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818409+0000", 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024941299999999998 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819512+0000", 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011029869999999999 2024-09-13T11:32:31.327 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.45790804299999999, 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.328 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.332 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:32:31.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:32:31.338 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1055 4.8 4:1d367c64:::smithi20438004-45:head [write 1515641~485017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.764504+0000", 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.45320243999999998, 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070551770000000001, 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:31.339 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1055 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.764504+0000", 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.764504+0000", 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:31.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.764505+0000", 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4339999999999999e-06 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.765405+0000", 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000900263 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:31.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.765418+0000", 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2928999999999999e-05 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.765427+0000", 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2150000000000007e-06 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.768952+0000", 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035248929999999999 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.769082+0000", 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012952100000000001 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.769194+0000", 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011247300000000001 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770858+0000", 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016640870000000001 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770932+0000", 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3195999999999995e-05 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.771502+0000", 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00057037600000000004 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.771531+0000", 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9057e-05 2024-09-13T11:32:31.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.771559+0000", 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7733e-05 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:31.345 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:31.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2649261 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:702: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:702: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:702: finishing write tid 5 to smithi20438004-23 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:702: oid 23 updating version 0 to 25 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:702: oid 23 updating version 25 to 28 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:702: oid 23 version 28 is already newer than 26 2024-09-13T11:32:31.348 INFO:tasks.rados.rados.0.smithi204.stdout:702: oid 23 version 28 is already newer than 27 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 28 (ObjNum 85 snap 0 seq_num 85) dirty exists 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:702: left oid 23 (ObjNum 85 snap 0 seq_num 85) 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:704: finishing write tid 1 to smithi20438004-12 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:704: finishing write tid 2 to smithi20438004-12 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:701: done (15 left) 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:702: done (14 left) 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:703: done (13 left) 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:717: write oid 43 current snap is 0 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:708: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2337392 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 842618 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:704: finishing write tid 3 to smithi20438004-12 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:704: finishing write tid 4 to smithi20438004-12 2024-09-13T11:32:31.349 INFO:tasks.rados.rados.0.smithi204.stdout:704: finishing write tid 5 to smithi20438004-12 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:704: oid 12 updating version 0 to 23 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:704: oid 12 version 23 is already newer than 21 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:704: oid 12 version 23 is already newer than 22 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:704: oid 12 updating version 23 to 24 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 24 (ObjNum 86 snap 0 seq_num 86) dirty exists 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:704: left oid 12 (ObjNum 86 snap 0 seq_num 86) 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:705: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:31.350 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:31.350 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:31.345+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:31.391 INFO:tasks.rados.rados.0.smithi204.stdout:707: expect (ObjNum 61 snap 0 seq_num 61) 2024-09-13T11:32:31.438 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:31.448 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:31.446+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:31.457 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:31.458 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:31.458 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:31.458 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:31.466 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:32:31.534 INFO:tasks.rados.rados.0.smithi204.stdout:709: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:31.541 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 969815 2024-09-13T11:32:31.545 INFO:teuthology.orchestra.run.smithi154.stderr:Exporting 4.4 info 4.4( v 39'34 (39'31,39'34] local-lis/les=31/32 n=3 ec=31/31 lis/c=31/31 les/c/f=32/32/0 sis=31) 2024-09-13T11:32:31.546 INFO:teuthology.orchestra.run.smithi154.stderr:Read #4:2451796c:::smithi20438004-17:head# 2024-09-13T11:32:31.548 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:31.547+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:31.551 INFO:teuthology.orchestra.run.smithi154.stderr:Read #4:28ee04de:::smithi20438004-15:head# 2024-09-13T11:32:31.556 INFO:teuthology.orchestra.run.smithi154.stderr:Read #4:2b094039:::smithi20438004-40:head# 2024-09-13T11:32:31.561 INFO:teuthology.orchestra.run.smithi154.stderr:Export successful 2024-09-13T11:32:31.575 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:31.576 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:31.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:32:31.649 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:31.648+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:31.671 INFO:tasks.rados.rados.0.smithi204.stdout:706: expect (ObjNum 69 snap 0 seq_num 69) 2024-09-13T11:32:31.672 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:31.691 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-09-13T11:32:31.691 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:31.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:32:31.751 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:31.749+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:31.757 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2527942 2024-09-13T11:32:31.807 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:31.808 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:31.808 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:31.808 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:31.817 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:32:31.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 4.4 2024-09-13T11:32:31.841 INFO:tasks.rados.rados.0.smithi204.stdout:710: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-13T11:32:31.851 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:31.850+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:31.924 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:31.924 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:31.925 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:31.925 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:31.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:32:31.976 INFO:tasks.rados.rados.0.smithi204.stdout:711: finishing set_chunk to oid 49 2024-09-13T11:32:31.977 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 770560 length: 270336 2024-09-13T11:32:31.977 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:31.977 INFO:tasks.rados.rados.0.smithi204.stdout:712: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:31.978 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:31.977+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:31.980 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 795238 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:714: finishing set_chunk to oid 8 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 500224 length: 1157120 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:715: finishing write tid 1 to smithi20438004-30 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:715: finishing write tid 2 to smithi20438004-30 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:715: finishing write tid 3 to smithi20438004-30 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:715: finishing write tid 4 to smithi20438004-30 2024-09-13T11:32:32.039 INFO:tasks.rados.rados.0.smithi204.stdout:715: finishing write tid 5 to smithi20438004-30 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:715: oid 30 updating version 0 to 32 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:715: oid 30 version 32 is already newer than 30 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:715: oid 30 version 32 is already newer than 31 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:715: oid 30 version 32 is already newer than 29 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 32 (ObjNum 87 snap 0 seq_num 87) dirty exists 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:715: left oid 30 (ObjNum 87 snap 0 seq_num 87) 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:716: finishing set_chunk to oid 24 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 84480 length: 2301952 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:32.040 INFO:tasks.rados.rados.0.smithi204.stdout:717: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:204 3.0 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:32.044 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.9576096610000002, 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025518434, 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 204 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084160+0000", 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.463e-06 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084171+0000", 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1206000000000001e-05 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084199+0000", 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7796e-05 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084215+0000", 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5820000000000001e-05 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084389+0000", 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174134 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084589+0000", 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020036899999999999 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.091157+0000", 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065682249999999996 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.096484+0000", 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053268400000000002 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.102639+0000", 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061551430000000001 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109591+0000", 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006951774 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109627+0000", 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6198999999999997e-05 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109676+0000", 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8464999999999998e-05 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:205 3.0 3:03052d75:::smithi20438004-6 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.957521377, 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.046448324999999999, 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.051 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 205 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8999999999999995e-07 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084251+0000", 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9899999999999997e-06 2024-09-13T11:32:32.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084262+0000", 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.023e-05 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084270+0000", 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0669999999999992e-06 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084746+0000", 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047623199999999999 2024-09-13T11:32:32.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084842+0000", 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6065999999999998e-05 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.094307+0000", 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094653670000000006 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.098690+0000", 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043829810000000002 2024-09-13T11:32:32.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.107503+0000", 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088122559999999992 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130597+0000", 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023094078000000001 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130642+0000", 2024-09-13T11:32:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5463999999999997e-05 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130694+0000", 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1703999999999998e-05 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:213 3.0 3:22c1b591:::smithi20438004-14 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.9573293329999997, 2024-09-13T11:32:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.055117410999999998, 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 213 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5200000000000002e-07 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084441+0000", 2024-09-13T11:32:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084447+0000", 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5430000000000001e-06 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084451+0000", 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5829999999999998e-06 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084997+0000", 2024-09-13T11:32:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054659399999999996 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085141+0000", 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014359400000000001 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.097495+0000", 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012354146999999999 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.099703+0000", 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022078079999999999 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.112967+0000", 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013264582 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139474+0000", 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026506980999999999 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139513+0000", 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8322999999999999e-05 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139555+0000", 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2475999999999998e-05 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:209 3.2 e38/30 3:6afe79ef:::smithi20438004-10 low_tier:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.090851+0000", 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.950915824, 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011996711, 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.090851+0000", 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.090851+0000", 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.090851+0000", 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9500000000000001e-07 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.100364+0000", 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095125079999999994 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.100376+0000", 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1790000000000001e-05 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.100386+0000", 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0762e-05 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.100448+0000", 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1339000000000003e-05 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.100464+0000", 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6446000000000001e-05 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.102813+0000", 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023487600000000001 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.102848+0000", 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4910999999999998e-05 2024-09-13T11:32:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:217 3.0 3:33b1e45a:::smithi20438004-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.9333068530000004, 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03827937, 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 217 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108461+0000", 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.392e-06 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108467+0000", 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8429999999999998e-06 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108478+0000", 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0243e-05 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108483+0000", 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8000000000000004e-06 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108545+0000", 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1909000000000006e-05 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108674+0000", 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012890699999999999 2024-09-13T11:32:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.120164+0000", 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011489868 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.122425+0000", 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002260649 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.125843+0000", 2024-09-13T11:32:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034186220000000001 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146587+0000", 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020743276000000001 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146664+0000", 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7330999999999995e-05 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146739+0000", 2024-09-13T11:32:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5530000000000004e-05 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:222 3.0 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.9224298370000001, 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033143700999999998, 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 222 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119339+0000", 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.452e-06 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119345+0000", 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8499999999999996e-06 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119358+0000", 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119364+0000", 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3219999999999998e-06 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119411+0000", 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6186999999999997e-05 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119552+0000", 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000141204 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.128873+0000", 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093210610000000003 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.131258+0000", 2024-09-13T11:32:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023855529999999999 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.136698+0000", 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005439854 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152381+0000", 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015683002000000001 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152419+0000", 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7561000000000002e-05 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152481+0000", 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2038000000000005e-05 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:225 3.0 3:10cf6cc6:::smithi20438004-26 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.9082033139999997, 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026005851, 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 225 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133565+0000", 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.052e-06 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133571+0000", 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9349999999999999e-06 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133580+0000", 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8530000000000004e-06 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133586+0000", 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9610000000000002e-06 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133632+0000", 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5427999999999999e-05 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133735+0000", 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102829 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147872+0000", 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014137244 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152657+0000", 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047851830000000001 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159315+0000", 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066580700000000003 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159445+0000", 2024-09-13T11:32:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130104 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159513+0000", 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7897000000000004e-05 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159569+0000", 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6295000000000002e-05 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:227 3.0 3:162d2d40:::smithi20438004-28 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.9020020019999997, 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028012617, 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139766+0000", 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:32:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139770+0000", 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0149999999999997e-06 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139779+0000", 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4460000000000007e-06 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139784+0000", 2024-09-13T11:32:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8779999999999997e-06 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139822+0000", 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7750999999999998e-05 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139934+0000", 2024-09-13T11:32:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112378 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.150743+0000", 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010808768999999999 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.154264+0000", 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035207160000000001 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.164397+0000", 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010133675999999999 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167661+0000", 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032640759999999999 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167728+0000", 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7018000000000001e-05 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167778+0000", 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9197999999999999e-05 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:232 3.0 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8942357589999999, 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032659146999999999, 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 232 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147533+0000", 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147539+0000", 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8850000000000004e-06 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147555+0000", 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6065e-05 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147568+0000", 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3236e-05 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152097+0000", 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045292199999999996 2024-09-13T11:32:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152211+0000", 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001141 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.162843+0000", 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010631982999999999 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167140+0000", 2024-09-13T11:32:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042969829999999999 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.174951+0000", 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078107159999999997 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180068+0000", 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051167369999999997 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180131+0000", 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3140999999999997e-05 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180190+0000", 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9438999999999998e-05 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:234 3.0 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8934544330000005, 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036893333, 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 234 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148314+0000", 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.871e-06 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148323+0000", 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5010000000000004e-06 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148334+0000", 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1280999999999999e-05 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148342+0000", 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152513+0000", 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041709370000000004 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152594+0000", 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0315999999999995e-05 2024-09-13T11:32:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165507+0000", 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012913051 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.168150+0000", 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026431200000000001 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179750+0000", 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01160015 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185129+0000", 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053793729999999998 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185171+0000", 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2073000000000002e-05 2024-09-13T11:32:32.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4456999999999999e-05 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:233 3.2 e38/30 3:53d0adee:::smithi20438004-34 low_tier:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8882380630000002, 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018647573000000001, 2024-09-13T11:32:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153530+0000", 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4920000000000001e-06 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169595+0000", 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016064407999999999 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169603+0000", 2024-09-13T11:32:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7250000000000003e-06 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169609+0000", 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3900000000000001e-06 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169669+0000", 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0052e-05 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169684+0000", 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4778e-05 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.172138+0000", 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024544110000000001 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.172176+0000", 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8316999999999999e-05 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:239 3.0 3:3c12287e:::smithi20438004-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8816872990000002, 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037773656000000003, 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 239 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160086+0000", 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6810000000000003e-06 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160097+0000", 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.101e-05 2024-09-13T11:32:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160103+0000", 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6319999999999996e-06 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160148+0000", 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.511e-05 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160285+0000", 2024-09-13T11:32:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000137508 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.175532+0000", 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015246902999999999 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180030+0000", 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044977300000000001 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185903+0000", 2024-09-13T11:32:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058733659999999997 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197739+0000", 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011835094000000001 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197801+0000", 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2226999999999999e-05 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197853+0000", 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2537999999999997e-05 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:241 3.0 3:2c9c5010:::smithi20438004-42 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8759177930000002, 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047296003000000003, 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 241 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165851+0000", 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5269999999999999e-06 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165867+0000", 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6827e-05 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165880+0000", 2024-09-13T11:32:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165890+0000", 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0101999999999999e-05 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167178+0000", 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00128834 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167324+0000", 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014554300000000001 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.178258+0000", 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010933951000000001 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180829+0000", 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025710849999999999 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.186910+0000", 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060810639999999997 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213073+0000", 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.02616305 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213112+0000", 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9292000000000003e-05 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213145+0000", 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3126000000000001e-05 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:240 3.2 e38/30 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8721320349999999, 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014316538, 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3599999999999999e-07 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181076+0000", 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011440584 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181086+0000", 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0138e-05 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181092+0000", 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8409999999999996e-06 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181142+0000", 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9694999999999998e-05 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181167+0000", 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5499000000000001e-05 2024-09-13T11:32:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.183914+0000", 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027467220000000001 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.183951+0000", 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7623000000000002e-05 2024-09-13T11:32:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:243 3.0 3:03808b00:::smithi20438004-44 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.864807152, 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042573002999999998, 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 243 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176961+0000", 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6610000000000001e-06 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176970+0000", 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2779999999999993e-06 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176982+0000", 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2711000000000001e-05 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176990+0000", 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000003e-06 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179794+0000", 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028036379999999998 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179911+0000", 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117095 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.195122+0000", 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015211107 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197868+0000", 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027461540000000002 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.204148+0000", 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006280435 2024-09-13T11:32:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219448+0000", 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015299787 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3766000000000001e-05 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219533+0000", 2024-09-13T11:32:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0880999999999997e-05 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:246 3.2 e38/30 3:68241bcb:::smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'6)", 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.187530+0000", 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8542369079999999, 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013176479, 2024-09-13T11:32:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.187530+0000", 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.187530+0000", 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.187531+0000", 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8500000000000001e-07 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197429+0000", 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098982489999999996 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197441+0000", 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2041999999999999e-05 2024-09-13T11:32:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197454+0000", 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3162e-05 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197525+0000", 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0802000000000004e-05 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197544+0000", 2024-09-13T11:32:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.931e-05 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.200660+0000", 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031153259999999999 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.200707+0000", 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7002999999999997e-05 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:251 4.6 4:6d3e8486:::smithi20438004-2:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8439216810000003, 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021332106999999999, 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 251 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197847+0000", 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197853+0000", 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0530000000000003e-06 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197864+0000", 2024-09-13T11:32:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0961999999999999e-05 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197874+0000", 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0377000000000001e-05 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197920+0000", 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5178e-05 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197996+0000", 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6373999999999998e-05 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218088+0000", 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020092445 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218692+0000", 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060317200000000002 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219008+0000", 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000316328 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219097+0000", 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8993000000000007e-05 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219140+0000", 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3167000000000002e-05 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219177+0000", 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7301000000000001e-05 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:253 4.6 4:600f3d3f:::smithi20438004-4:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8422151769999999, 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022588031000000001, 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199553+0000", 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9299999999999996e-07 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199557+0000", 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6110000000000004e-06 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199568+0000", 2024-09-13T11:32:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0582e-05 2024-09-13T11:32:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199574+0000", 2024-09-13T11:32:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3500000000000002e-06 2024-09-13T11:32:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199623+0000", 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8736999999999998e-05 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199673+0000", 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0232999999999998e-05 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219425+0000", 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019751614000000001 2024-09-13T11:32:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219918+0000", 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00049309600000000005 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221826+0000", 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019085479999999999 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222084+0000", 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025730000000000002 2024-09-13T11:32:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222114+0000", 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0048000000000001e-05 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222140+0000", 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6018999999999999e-05 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.10.22:1 3.0 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8384913379999999, 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016452626000000001, 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203277+0000", 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-06 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203282+0000", 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.882e-06 2024-09-13T11:32:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203290+0000", 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203300+0000", 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0162e-05 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203350+0000", 2024-09-13T11:32:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9487999999999998e-05 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203422+0000", 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2537e-05 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203577+0000", 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015495999999999999 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.204164+0000", 2024-09-13T11:32:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058732999999999995 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.205037+0000", 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087292200000000004 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219676+0000", 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014638463000000001 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219704+0000", 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8396999999999999e-05 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219728+0000", 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3944e-05 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:247 3.2 e38/30 3:4f19c0b8:::smithi20438004-48 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211473+0000", 2024-09-13T11:32:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8302940119999995, 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011972246000000001, 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211473+0000", 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211473+0000", 2024-09-13T11:32:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211473+0000", 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5899999999999996e-07 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220313+0000", 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088392499999999999 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220321+0000", 2024-09-13T11:32:32.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7490000000000004e-06 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220329+0000", 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4209999999999998e-06 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220383+0000", 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4339000000000003e-05 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:32.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220404+0000", 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0706000000000001e-05 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.223406+0000", 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030019259999999998 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.223445+0000", 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9295999999999998e-05 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.166 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:32.166 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:32.166 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:32.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:32:32.170 INFO:tasks.rados.rados.0.smithi204.stdout:717: writing smithi20438004-43 from 679016 to 1380811 tid 1 2024-09-13T11:32:32.170 INFO:tasks.rados.rados.0.smithi204.stdout:717: writing smithi20438004-43 from 1866533 to 2362557 tid 2 2024-09-13T11:32:32.170 INFO:tasks.rados.rados.0.smithi204.stdout:717: writing smithi20438004-43 from 2900635 to 3303278 tid 3 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:704: done (13 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:705: done (12 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:706: done (11 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:707: done (10 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:708: done (9 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:709: done (8 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:710: done (7 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:711: done (6 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:712: done (5 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:713: done (4 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:714: done (3 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:715: done (2 left) 2024-09-13T11:32:32.171 INFO:tasks.rados.rados.0.smithi204.stdout:716: done (1 left) 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:718: read oid 40 snap -1 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:718: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:719: read oid 14 snap -1 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:719: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:720: read oid 21 snap -1 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:720: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:721: read oid 27 snap -1 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:721: expect (ObjNum 58 snap 0 seq_num 58) 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:722: tier_evict oid 19 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:722: tier_evict oid 19 snap -1 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:717: finishing write tid 1 to smithi20438004-43 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:723: read oid 26 snap -1 2024-09-13T11:32:32.172 INFO:tasks.rados.rados.0.smithi204.stdout:723: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:724: read oid 8 snap -1 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:724: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:725: read oid 39 snap -1 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:725: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:726: read oid 47 snap -1 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:726: expect (ObjNum 61 snap 0 seq_num 61) 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:727: delete oid 33 current snap is 0 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:717: finishing write tid 2 to smithi20438004-43 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:727: done (10 left) 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:728: delete oid 3 current snap is 0 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:723: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1493795 2024-09-13T11:32:32.173 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2966273 2024-09-13T11:32:32.174 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:32.166+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:32.178 INFO:tasks.rados.rados.0.smithi204.stdout:720: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-13T11:32:32.179 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.233 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 401769 2024-09-13T11:32:32.256 INFO:tasks.rados.rados.0.smithi204.stdout:724: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:32.256 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.268 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:32.267+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:32.291 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:32.292 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:32.292 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:32.292 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:32.298 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.300 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:32:32.322 INFO:tasks.rados.rados.0.smithi204.stdout:719: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-13T11:32:32.325 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1601947 2024-09-13T11:32:32.369 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:32.367+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:32.376 INFO:tasks.rados.rados.0.smithi204.stdout:726: expect (ObjNum 61 snap 0 seq_num 61) 2024-09-13T11:32:32.404 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:32.404 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:32.404 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:32.404 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:32.405 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:32.409 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.413 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:32:32.469 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:32.468+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:32.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:32:32.505 INFO:tasks.rados.rados.0.smithi204.stdout:717: finishing write tid 3 to smithi20438004-43 2024-09-13T11:32:32.505 INFO:tasks.rados.rados.0.smithi204.stdout:717: finishing write tid 4 to smithi20438004-43 2024-09-13T11:32:32.505 INFO:tasks.rados.rados.0.smithi204.stdout:719: done (10 left) 2024-09-13T11:32:32.505 INFO:tasks.rados.rados.0.smithi204.stdout:720: done (9 left) 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:723: done (8 left) 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:724: done (7 left) 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:726: done (6 left) 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:728: done (5 left) 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:729: read oid 41 snap -1 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:729: expect (ObjNum 76 snap 0 seq_num 76) 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:730: delete oid 16 current snap is 0 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:717: finishing write tid 5 to smithi20438004-43 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:717: oid 43 updating version 0 to 50 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:717: oid 43 updating version 50 to 52 2024-09-13T11:32:32.506 INFO:tasks.rados.rados.0.smithi204.stdout:717: oid 43 version 52 is already newer than 49 2024-09-13T11:32:32.507 INFO:tasks.rados.rados.0.smithi204.stdout:717: oid 43 version 52 is already newer than 51 2024-09-13T11:32:32.507 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 52 (ObjNum 88 snap 0 seq_num 88) dirty exists 2024-09-13T11:32:32.507 INFO:tasks.rados.rados.0.smithi204.stdout:717: left oid 43 (ObjNum 88 snap 0 seq_num 88) 2024-09-13T11:32:32.507 INFO:tasks.rados.rados.0.smithi204.stdout:721: expect (ObjNum 58 snap 0 seq_num 58) 2024-09-13T11:32:32.522 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.4289875409999997, 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011006991000000001, 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.523 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091279+0000", 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.091e-06 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098180+0000", 2024-09-13T11:32:32.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069015359999999998 2024-09-13T11:32:32.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098194+0000", 2024-09-13T11:32:32.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.323e-05 2024-09-13T11:32:32.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098202+0000", 2024-09-13T11:32:32.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2879999999999993e-06 2024-09-13T11:32:32.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098267+0000", 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4733999999999994e-05 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098296+0000", 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102225+0000", 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039281940000000003 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102285+0000", 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3721820520000003, 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010863593, 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148085+0000", 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2810000000000001e-06 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153254+0000", 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051698430000000004 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153266+0000", 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1973e-05 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153274+0000", 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153331+0000", 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6492000000000001e-05 2024-09-13T11:32:32.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153396+0000", 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5083e-05 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158906+0000", 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055099570000000002 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158947+0000", 2024-09-13T11:32:32.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3669695060000002, 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096623879999999992, 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157238+0000", 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039414139999999999 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157253+0000", 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5566e-05 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157260+0000", 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.477e-06 2024-09-13T11:32:32.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157315+0000", 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4857e-05 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157341+0000", 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:32:32.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162899+0000", 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055583480000000003 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162958+0000", 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9169000000000002e-05 2024-09-13T11:32:32.538 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3573114210000004, 2024-09-13T11:32:32.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011410943999999999, 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162955+0000", 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0669999999999999e-06 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167312+0000", 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004357072 2024-09-13T11:32:32.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167333+0000", 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0842000000000001e-05 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167341+0000", 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4430000000000003e-06 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167406+0000", 2024-09-13T11:32:32.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4245000000000006e-05 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167433+0000", 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7500000000000001e-05 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174280+0000", 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068473969999999999 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174365+0000", 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4377999999999997e-05 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3444836579999997, 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099135219999999993, 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.544 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175783+0000", 2024-09-13T11:32:32.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179929+0000", 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041461989999999997 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179943+0000", 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4731e-05 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179953+0000", 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180049+0000", 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6909000000000003e-05 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180073+0000", 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4077e-05 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185641+0000", 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055674330000000001 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185695+0000", 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4273e-05 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.548 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:258 4.3 4:c717f8db:::smithi20438004-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3067163920000002, 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017162947000000001, 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 258 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213551+0000", 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0760000000000001e-06 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213566+0000", 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5441999999999999e-05 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213585+0000", 2024-09-13T11:32:32.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8623000000000001e-05 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213600+0000", 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4783999999999999e-05 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213667+0000", 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213747+0000", 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9794000000000002e-05 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228488+0000", 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014740964 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229259+0000", 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077144599999999996 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229743+0000", 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048437000000000003 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230652+0000", 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090904600000000005 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230688+0000", 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5474e-05 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230712+0000", 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3954000000000001e-05 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:259 4.1 4:8533c921:::smithi20438004-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3024407900000003, 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014548484, 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.555 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 259 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:32:32.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217826+0000", 2024-09-13T11:32:32.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6360000000000001e-06 2024-09-13T11:32:32.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217833+0000", 2024-09-13T11:32:32.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8700000000000003e-06 2024-09-13T11:32:32.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217844+0000", 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217851+0000", 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217896+0000", 2024-09-13T11:32:32.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4808000000000002e-05 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217954+0000", 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7856000000000003e-05 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229813+0000", 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011859533 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231353+0000", 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001539503 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232188+0000", 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083500600000000001 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232284+0000", 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6645999999999996e-05 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232325+0000", 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0071999999999999e-05 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232373+0000", 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8498999999999998e-05 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:266 4.4 4:2451796c:::smithi20438004-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.567 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2979409940000002, 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099464859999999992, 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 266 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222325+0000", 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222330+0000", 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999997e-06 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222341+0000", 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0528e-05 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222347+0000", 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6409999999999996e-06 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222406+0000", 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8449000000000002e-05 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222487+0000", 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1167000000000004e-05 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230197+0000", 2024-09-13T11:32:32.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077105120000000001 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230937+0000", 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000740159 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231296+0000", 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035862399999999999 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232186+0000", 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089002899999999997 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232234+0000", 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7414999999999999e-05 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232271+0000", 2024-09-13T11:32:32.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7323000000000002e-05 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:271 4.4 4:2c957d44:::smithi20438004-22:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.227296+0000", 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2929693929999999, 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009342408, 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.574 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 271 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227296+0000", 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227296+0000", 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227297+0000", 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2309999999999999e-06 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227305+0000", 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5290000000000003e-06 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227343+0000", 2024-09-13T11:32:32.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8679e-05 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227351+0000", 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3050000000000004e-06 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227409+0000", 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8022000000000001e-05 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227484+0000", 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5109000000000002e-05 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233193+0000", 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057090370000000001 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233779+0000", 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00058593299999999996 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234281+0000", 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050244099999999996 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236547+0000", 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002266002 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236599+0000", 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2086000000000002e-05 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236638+0000", 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9033999999999999e-05 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:283 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.580 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2868128590000003, 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011861159, 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 283 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233453+0000", 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233457+0000", 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1150000000000004e-06 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233465+0000", 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233471+0000", 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3450000000000002e-06 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233526+0000", 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4790000000000002e-05 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233602+0000", 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6253000000000003e-05 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237634+0000", 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040318580000000001 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238163+0000", 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052849899999999998 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242711+0000", 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045478530000000001 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245235+0000", 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025239160000000002 2024-09-13T11:32:32.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245286+0000", 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1739e-05 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245314+0000", 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7280999999999999e-05 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.586 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:284 4.1 4:8b052c81:::smithi20438004-35:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2864628470000001, 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011653294, 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 284 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233803+0000", 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4600000000000003e-07 2024-09-13T11:32:32.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233808+0000", 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233817+0000", 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9870000000000005e-06 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233824+0000", 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8290000000000002e-06 2024-09-13T11:32:32.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233874+0000", 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9877999999999999e-05 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233943+0000", 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238024+0000", 2024-09-13T11:32:32.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040810350000000002 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238440+0000", 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041507799999999999 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242785+0000", 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043457890000000001 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245363+0000", 2024-09-13T11:32:32.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025773359999999999 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245416+0000", 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3156999999999999e-05 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245456+0000", 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0000000000000003e-05 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:286 4.5 4:a9b0c9bd:::smithi20438004-37:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2854810859999999, 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013081485, 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 286 2024-09-13T11:32:32.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234785+0000", 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234790+0000", 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.651e-06 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234800+0000", 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234807+0000", 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.861e-06 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234868+0000", 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0582999999999999e-05 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234967+0000", 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054743730000000003 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241205+0000", 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076331700000000001 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241351+0000", 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619399999999999 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247766+0000", 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064151290000000003 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247825+0000", 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8739999999999997e-05 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247866+0000", 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747000000000001e-05 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.598 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:290 4.1 4:8d6fdfdb:::smithi20438004-41:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2809269370000003, 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014414148999999999, 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 290 2024-09-13T11:32:32.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239339+0000", 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239344+0000", 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2290000000000002e-06 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239354+0000", 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239361+0000", 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.092e-06 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239406+0000", 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126000000000001e-05 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239452+0000", 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6192000000000001e-05 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248478+0000", 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090262209999999992 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249309+0000", 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083057300000000003 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253400+0000", 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040908760000000002 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253662+0000", 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026229500000000001 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253715+0000", 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2836e-05 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253753+0000", 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7526e-05 2024-09-13T11:32:32.604 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:292 4.1 4:84fa1f64:::smithi20438004-43:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2805037239999999, 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010951834000000001, 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.605 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 292 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8800000000000002e-07 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239765+0000", 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239773+0000", 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8229999999999995e-06 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239779+0000", 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9900000000000002e-06 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239831+0000", 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2326e-05 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239910+0000", 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8623999999999998e-05 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245574+0000", 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005663927 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.246099+0000", 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052478300000000002 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247963+0000", 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001864166 2024-09-13T11:32:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250625+0000", 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002662228 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250676+0000", 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0989000000000003e-05 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250713+0000", 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7360000000000001e-05 2024-09-13T11:32:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:296 4.5 4:a5f8f9e2:::smithi20438004-47:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.241972+0000", 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2782935809999998, 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090154020000000005, 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 296 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241972+0000", 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241972+0000", 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241973+0000", 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1039999999999999e-06 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241978+0000", 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1590000000000001e-06 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241988+0000", 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0358e-05 2024-09-13T11:32:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241995+0000", 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1040000000000001e-06 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242082+0000", 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7004999999999998e-05 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242158+0000", 2024-09-13T11:32:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5668999999999997e-05 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248884+0000", 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067260699999999998 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249634+0000", 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074974099999999995 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249841+0000", 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020693799999999999 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250898+0000", 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010574409999999999 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250942+0000", 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3195999999999998e-05 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250987+0000", 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5617e-05 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:317 4.1 4:8c30e60e:::smithi20438004-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2693798190000001, 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013134860999999999, 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 317 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250892+0000", 2024-09-13T11:32:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250900+0000", 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250908+0000", 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.337e-06 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250967+0000", 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8201000000000001e-05 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251045+0000", 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299000000000004e-05 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258769+0000", 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007724323 2024-09-13T11:32:32.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259599+0000", 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082958200000000004 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260992+0000", 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00139291 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263918+0000", 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029263190000000001 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263971+0000", 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2670999999999998e-05 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264020+0000", 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.623 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:330 4.3 4:c7f26bfe:::smithi20438004-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2681570039999999, 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011943621, 2024-09-13T11:32:32.624 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 330 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252109+0000", 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252116+0000", 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9199999999999998e-06 2024-09-13T11:32:32.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252129+0000", 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2768e-05 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252138+0000", 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5939999999999994e-06 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252205+0000", 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:32:32.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252280+0000", 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4930000000000003e-05 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259484+0000", 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00720434 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259994+0000", 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051054200000000001 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260333+0000", 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033855000000000002 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263947+0000", 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00361452 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264015+0000", 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7155000000000003e-05 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264052+0000", 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305000000000003e-05 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:333 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.630 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.267866808, 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011814712, 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 333 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:32.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252406+0000", 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252416+0000", 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-05 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252423+0000", 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252486+0000", 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2805999999999995e-05 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252540+0000", 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3897000000000003e-05 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259138+0000", 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065977429999999997 2024-09-13T11:32:32.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259785+0000", 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064727700000000003 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264056+0000", 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042706710000000002 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264152+0000", 2024-09-13T11:32:32.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5877000000000004e-05 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264180+0000", 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8674999999999999e-05 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264213+0000", 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3015000000000002e-05 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:433 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2433159610000004, 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098551839999999995, 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-13T11:32:32.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276950+0000", 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999995e-07 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276955+0000", 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1059999999999999e-06 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276965+0000", 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:32:32.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276977+0000", 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1518000000000001e-05 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277035+0000", 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8133000000000001e-05 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277125+0000", 2024-09-13T11:32:32.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0148999999999994e-05 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282235+0000", 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005109461 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282725+0000", 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049045700000000002 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286634+0000", 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039087929999999998 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286707+0000", 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2692000000000001e-05 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286772+0000", 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5828999999999996e-05 2024-09-13T11:32:32.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286805+0000", 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2134000000000001e-05 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.643 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:791 4.5 4:a5f8f9e2:::smithi20438004-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.8204330510000002, 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019917450999999999, 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 791 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:32.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699834+0000", 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-09-13T11:32:32.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699840+0000", 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5989999999999998e-06 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699852+0000", 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699861+0000", 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5259999999999993e-06 2024-09-13T11:32:32.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699949+0000", 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8083000000000004e-05 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699953+0000", 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3869999999999997e-06 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701189+0000", 2024-09-13T11:32:32.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012354740000000001 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701250+0000", 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1733e-05 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:32.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.719750+0000", 2024-09-13T11:32:32.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018499504 2024-09-13T11:32:32.649 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.649 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.649 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.649 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:32.649 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:32.649 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:32.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:32.651 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:32.649+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:32.652 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 80146 2024-09-13T11:32:32.652 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.652 INFO:tasks.rados.rados.0.smithi204.stdout:722: got (22) Invalid argument 2024-09-13T11:32:32.652 INFO:tasks.rados.rados.0.smithi204.stdout:725: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:32.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:32:32.693 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.751 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:32.750+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:32.757 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.777 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:32.795 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:32.795 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:32.796 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:32.796 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:32.796 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:32.796 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:32.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:32:32.812 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:32.812 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:32.822 INFO:tasks.rados.rados.0.smithi204.stdout:718: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:32.822 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.828 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:32.828 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:32.845 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:32.845 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:32.852 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:32.850+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:32.862 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:32.863 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:32.880 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:32.880 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:32.896 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:32.908 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:32.908 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:32.914 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:32.914 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:32.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:32:32.931 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:32.931 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:32.943 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2541837 2024-09-13T11:32:32.948 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:32.948 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:32.956 INFO:tasks.rados.rados.0.smithi204.stdout:729: expect (ObjNum 76 snap 0 seq_num 76) 2024-09-13T11:32:32.959 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:32.957+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:32.962 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1025222 2024-09-13T11:32:32.963 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:32.964 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:32.981 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:32.981 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:32.985 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:32.998 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:32.998 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:33.015 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:33.015 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:33.019 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:33.036 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2806730 2024-09-13T11:32:33.041 INFO:tasks.rados.rados.0.smithi204.stdout:717: done (6 left) 2024-09-13T11:32:33.041 INFO:tasks.rados.rados.0.smithi204.stdout:718: done (5 left) 2024-09-13T11:32:33.041 INFO:tasks.rados.rados.0.smithi204.stdout:721: done (4 left) 2024-09-13T11:32:33.041 INFO:tasks.rados.rados.0.smithi204.stdout:722: done (3 left) 2024-09-13T11:32:33.041 INFO:tasks.rados.rados.0.smithi204.stdout:725: done (2 left) 2024-09-13T11:32:33.041 INFO:tasks.rados.rados.0.smithi204.stdout:729: done (1 left) 2024-09-13T11:32:33.042 INFO:tasks.rados.rados.0.smithi204.stdout:730: done (0 left) 2024-09-13T11:32:33.042 INFO:tasks.rados.rados.0.smithi204.stdout:731: tier_flush oid 8 2024-09-13T11:32:33.042 INFO:tasks.rados.rados.0.smithi204.stdout:731: tier_flush oid 8 snap -1 2024-09-13T11:32:33.042 INFO:tasks.rados.rados.0.smithi204.stdout:732: write (excl) oid 43 current snap is 0 2024-09-13T11:32:33.042 INFO:tasks.rados.rados.0.smithi204.stdout:732: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-13T11:32:33.048 INFO:tasks.rados.rados.0.smithi204.stdout:732: writing smithi20438004-43 from 746244 to 1465388 tid 1 2024-09-13T11:32:33.053 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:33.052+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:33.054 INFO:tasks.rados.rados.0.smithi204.stdout:732: writing smithi20438004-43 from 2100461 to 2603823 tid 2 2024-09-13T11:32:33.060 INFO:tasks.rados.rados.0.smithi204.stdout:732: writing smithi20438004-43 from 3126680 to 3546244 tid 3 2024-09-13T11:32:33.060 INFO:tasks.rados.rados.0.smithi204.stdout:733: read oid 3 snap -1 2024-09-13T11:32:33.060 INFO:tasks.rados.rados.0.smithi204.stdout:733: expect deleted 2024-09-13T11:32:33.060 INFO:tasks.rados.rados.0.smithi204.stdout:734: read oid 25 snap -1 2024-09-13T11:32:33.060 INFO:tasks.rados.rados.0.smithi204.stdout:734: expect (ObjNum 59 snap 0 seq_num 59) 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:735: read oid 34 snap -1 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:735: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:736: read oid 1 snap -1 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:736: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:737: write (excl) oid 30 current snap is 0 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:731: got (0) Success 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:732: finishing write tid 1 to smithi20438004-43 2024-09-13T11:32:33.061 INFO:tasks.rados.rados.0.smithi204.stdout:737: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-13T11:32:33.063 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.063 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.069 INFO:tasks.rados.rados.0.smithi204.stdout:737: writing smithi20438004-30 from 763051 to 1557444 tid 1 2024-09-13T11:32:33.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:32:33.076 INFO:tasks.rados.rados.0.smithi204.stdout:737: writing smithi20438004-30 from 2158943 to 2805963 tid 2 2024-09-13T11:32:33.080 INFO:tasks.rados.rados.0.smithi204.stdout:737: writing smithi20438004-30 from 3437750 to 3563051 tid 3 2024-09-13T11:32:33.080 INFO:tasks.rados.rados.0.smithi204.stdout:731: done (6 left) 2024-09-13T11:32:33.080 INFO:tasks.rados.rados.0.smithi204.stdout:738: read oid 13 snap -1 2024-09-13T11:32:33.080 INFO:tasks.rados.rados.0.smithi204.stdout:738: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T11:32:33.080 INFO:tasks.rados.rados.0.smithi204.stdout:739: write oid 27 current snap is 0 2024-09-13T11:32:33.080 INFO:tasks.rados.rados.0.smithi204.stdout:732: finishing write tid 2 to smithi20438004-43 2024-09-13T11:32:33.080 INFO:tasks.rados.rados.0.smithi204.stdout:739: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-09-13T11:32:33.087 INFO:tasks.rados.rados.0.smithi204.stdout:739: writing smithi20438004-27 from 779858 to 1249500 tid 1 2024-09-13T11:32:33.093 INFO:tasks.rados.rados.0.smithi204.stdout:739: writing smithi20438004-27 from 1817425 to 2324456 tid 2 2024-09-13T11:32:33.099 INFO:tasks.rados.rados.0.smithi204.stdout:739: writing smithi20438004-27 from 2781526 to 3368154 tid 3 2024-09-13T11:32:33.099 INFO:tasks.rados.rados.0.smithi204.stdout:740: read oid 23 snap -1 2024-09-13T11:32:33.099 INFO:tasks.rados.rados.0.smithi204.stdout:740: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-13T11:32:33.099 INFO:tasks.rados.rados.0.smithi204.stdout:741: read oid 47 snap -1 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:741: expect (ObjNum 61 snap 0 seq_num 61) 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:742: read oid 6 snap -1 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:742: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:743: read oid 28 snap -1 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:743: expect deleted 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:744: read oid 37 snap -1 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:744: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:745: read oid 41 snap -1 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:745: expect (ObjNum 76 snap 0 seq_num 76) 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:746: tier_flush oid 48 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:746: tier_flush oid 48 snap -1 2024-09-13T11:32:33.100 INFO:tasks.rados.rados.0.smithi204.stdout:747: set_chunk oid 18 target oid 2024-09-13T11:32:33.115 INFO:tasks.rados.rados.0.smithi204.stdout:747: set_chunk oid 18 offset: 1851392 length: 71680 target oid d906d767648a81564b9bd8689409cad523efb1f1b085d85ff04cda86dc529034 offset: 0 snap -1 2024-09-13T11:32:33.116 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:33.116 INFO:tasks.rados.rados.0.smithi204.stdout:736: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-13T11:32:33.154 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:33.153+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:33.164 INFO:tasks.rados.rados.0.smithi204.stdout:735: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T11:32:33.183 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.184 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.184 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1476170 2024-09-13T11:32:33.192 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:32:33.223 INFO:teuthology.orchestra.run.smithi154.stdout: marking collection for removal 2024-09-13T11:32:33.224 INFO:teuthology.orchestra.run.smithi154.stdout:setting '_remove' omap key 2024-09-13T11:32:33.224 INFO:tasks.rados.rados.0.smithi204.stdout:732: finishing write tid 3 to smithi20438004-43 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:732: finishing write tid 4 to smithi20438004-43 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:732: finishing write tid 5 to smithi20438004-43 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:732: oid 43 updating version 0 to 54 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:732: oid 43 version 54 is already newer than 53 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:732: oid 43 updating version 54 to 56 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:732: oid 43 version 56 is already newer than 55 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 56 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:732: left oid 43 (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:33.225 INFO:tasks.rados.rados.0.smithi204.stdout:734: expect (ObjNum 59 snap 0 seq_num 59) 2024-09-13T11:32:33.226 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:33.226 INFO:teuthology.orchestra.run.smithi154.stdout:finish_remove_pgs 4.4_head removing 4.4 2024-09-13T11:32:33.227 INFO:teuthology.orchestra.run.smithi154.stdout:Remove successful 2024-09-13T11:32:33.255 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:33.253+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:33.294 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.294 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.295 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2857077 2024-09-13T11:32:33.296 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:33.302 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:32:33.340 INFO:tasks.rados.rados.0.smithi204.stdout:737: finishing write tid 1 to smithi20438004-30 2024-09-13T11:32:33.340 INFO:tasks.rados.rados.0.smithi204.stdout:737: finishing write tid 2 to smithi20438004-30 2024-09-13T11:32:33.340 INFO:tasks.rados.rados.0.smithi204.stdout:738: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-13T11:32:33.340 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 504929 2024-09-13T11:32:33.355 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:33.354+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:33.359 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 444291 2024-09-13T11:32:33.404 INFO:tasks.rados.rados.0.smithi204.stdout:737: finishing write tid 3 to smithi20438004-30 2024-09-13T11:32:33.404 INFO:tasks.rados.rados.0.smithi204.stdout:737: finishing write tid 4 to smithi20438004-30 2024-09-13T11:32:33.404 INFO:tasks.rados.rados.0.smithi204.stdout:737: finishing write tid 5 to smithi20438004-30 2024-09-13T11:32:33.404 INFO:tasks.rados.rados.0.smithi204.stdout:737: oid 30 updating version 0 to 34 2024-09-13T11:32:33.404 INFO:tasks.rados.rados.0.smithi204.stdout:737: oid 30 version 34 is already newer than 33 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:737: oid 30 updating version 34 to 35 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:737: oid 30 updating version 35 to 36 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 36 (ObjNum 93 snap 0 seq_num 93) dirty exists 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:737: left oid 30 (ObjNum 93 snap 0 seq_num 93) 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:732: done (15 left) 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:733: done (14 left) 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:734: done (13 left) 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:735: done (12 left) 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:736: done (11 left) 2024-09-13T11:32:33.405 INFO:tasks.rados.rados.0.smithi204.stdout:737: done (10 left) 2024-09-13T11:32:33.406 INFO:tasks.rados.rados.0.smithi204.stdout:738: done (9 left) 2024-09-13T11:32:33.406 INFO:tasks.rados.rados.0.smithi204.stdout:748: write oid 31 current snap is 0 2024-09-13T11:32:33.406 INFO:tasks.rados.rados.0.smithi204.stdout:739: finishing write tid 1 to smithi20438004-27 2024-09-13T11:32:33.406 INFO:tasks.rados.rados.0.smithi204.stdout:739: finishing write tid 2 to smithi20438004-27 2024-09-13T11:32:33.406 INFO:tasks.rados.rados.0.smithi204.stdout:748: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-13T11:32:33.406 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.406 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.410 INFO:tasks.rados.rados.0.smithi204.stdout:748: writing smithi20438004-31 from 796665 to 1341556 tid 1 2024-09-13T11:32:33.414 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:32:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:748: writing smithi20438004-31 from 1759554 to 2410243 tid 2 2024-09-13T11:32:33.423 INFO:tasks.rados.rados.0.smithi204.stdout:748: writing smithi20438004-31 from 2976243 to 3596665 tid 3 2024-09-13T11:32:33.423 INFO:tasks.rados.rados.0.smithi204.stdout:749: read oid 46 snap -1 2024-09-13T11:32:33.423 INFO:tasks.rados.rados.0.smithi204.stdout:749: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T11:32:33.423 INFO:tasks.rados.rados.0.smithi204.stdout:750: write oid 19 current snap is 0 2024-09-13T11:32:33.423 INFO:tasks.rados.rados.0.smithi204.stdout:750: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-13T11:32:33.428 INFO:tasks.rados.rados.0.smithi204.stdout:750: writing smithi20438004-19 from 413472 to 1033612 tid 1 2024-09-13T11:32:33.435 INFO:tasks.rados.rados.0.smithi204.stdout:750: writing smithi20438004-19 from 1818036 to 2328736 tid 2 2024-09-13T11:32:33.442 INFO:tasks.rados.rados.0.smithi204.stdout:750: writing smithi20438004-19 from 3120019 to 3613472 tid 3 2024-09-13T11:32:33.443 INFO:tasks.rados.rados.0.smithi204.stdout:751: write oid 22 current snap is 0 2024-09-13T11:32:33.443 INFO:tasks.rados.rados.0.smithi204.stdout:751: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-13T11:32:33.448 INFO:tasks.rados.rados.0.smithi204.stdout:751: writing smithi20438004-22 from 430279 to 1125668 tid 1 2024-09-13T11:32:33.455 INFO:tasks.rados.rados.0.smithi204.stdout:751: writing smithi20438004-22 from 1876518 to 2530876 tid 2 2024-09-13T11:32:33.458 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:33.457+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:33.461 INFO:tasks.rados.rados.0.smithi204.stdout:751: writing smithi20438004-22 from 3031089 to 3567886 tid 3 2024-09-13T11:32:33.461 INFO:tasks.rados.rados.0.smithi204.stdout:752: tier_promote oid 35 2024-09-13T11:32:33.461 INFO:tasks.rados.rados.0.smithi204.stdout:753: tier_evict oid 2 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:753: tier_evict oid 2 snap -1 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:739: finishing write tid 3 to smithi20438004-27 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:739: finishing write tid 4 to smithi20438004-27 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:739: finishing write tid 5 to smithi20438004-27 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:739: oid 27 updating version 0 to 49 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:739: oid 27 version 49 is already newer than 48 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:739: oid 27 updating version 49 to 50 2024-09-13T11:32:33.462 INFO:tasks.rados.rados.0.smithi204.stdout:739: oid 27 updating version 50 to 51 2024-09-13T11:32:33.463 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 51 (ObjNum 94 snap 0 seq_num 94) dirty exists 2024-09-13T11:32:33.463 INFO:tasks.rados.rados.0.smithi204.stdout:739: left oid 27 (ObjNum 94 snap 0 seq_num 94) 2024-09-13T11:32:33.463 INFO:tasks.rados.rados.0.smithi204.stdout:742: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:33.480 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:33.497 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:32:33.505 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:33.523 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.523 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.530 INFO:tasks.rados.rados.0.smithi204.stdout:740: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-13T11:32:33.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:32:33.557 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:33.556+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:33.607 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2133828 2024-09-13T11:32:33.658 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:33.658+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:33.674 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.674 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:32:33.707 INFO:tasks.rados.rados.0.smithi204.stdout:741: expect (ObjNum 61 snap 0 seq_num 61) 2024-09-13T11:32:33.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.4.2 2024-09-13T11:32:33.759 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:33.758+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:33.789 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1004064 2024-09-13T11:32:33.789 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.789 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:32:33.822 INFO:tasks.rados.rados.0.smithi204.stdout:745: expect (ObjNum 76 snap 0 seq_num 76) 2024-09-13T11:32:33.823 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:33.858 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:33.860 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:33.859+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:33.921 INFO:tasks.rados.rados.0.smithi204.stdout:744: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-13T11:32:33.929 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:33.930 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:33.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:32:33.961 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:33.960+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:33.975 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1493109 2024-09-13T11:32:34.012 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1116163 2024-09-13T11:32:34.042 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.042 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.050 INFO:tasks.rados.rados.0.smithi204.stdout:747: finishing set_chunk to oid 18 2024-09-13T11:32:34.050 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 43 (ObjNum 81 snap 0 seq_num 81) dirty exists 2024-09-13T11:32:34.050 INFO:tasks.rados.rados.0.smithi204.stdout:746: got (0) Success 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 79 snap 0 seq_num 79) dirty exists 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:739: done (14 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:740: done (13 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:741: done (12 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:742: done (11 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:743: done (10 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:744: done (9 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:745: done (8 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:746: done (7 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:747: done (6 left) 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:754: write oid 17 current snap is 0 2024-09-13T11:32:34.051 INFO:tasks.rados.rados.0.smithi204.stdout:754: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-13T11:32:34.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:32:34.056 INFO:tasks.rados.rados.0.smithi204.stdout:754: writing smithi20438004-17 from 447086 to 1217724 tid 1 2024-09-13T11:32:34.061 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:34.061+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:34.063 INFO:tasks.rados.rados.0.smithi204.stdout:754: writing smithi20438004-17 from 1935000 to 2733016 tid 2 2024-09-13T11:32:34.068 INFO:tasks.rados.rados.0.smithi204.stdout:754: writing smithi20438004-17 from 3458512 to 3647086 tid 3 2024-09-13T11:32:34.068 INFO:tasks.rados.rados.0.smithi204.stdout:755: write (excl) oid 14 current snap is 0 2024-09-13T11:32:34.068 INFO:tasks.rados.rados.0.smithi204.stdout:755: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-13T11:32:34.073 INFO:tasks.rados.rados.0.smithi204.stdout:755: writing smithi20438004-14 from 463893 to 1026133 tid 1 2024-09-13T11:32:34.080 INFO:tasks.rados.rados.0.smithi204.stdout:755: writing smithi20438004-14 from 1593482 to 2251509 tid 2 2024-09-13T11:32:34.086 INFO:tasks.rados.rados.0.smithi204.stdout:755: writing smithi20438004-14 from 2685935 to 3245276 tid 3 2024-09-13T11:32:34.086 INFO:tasks.rados.rados.0.smithi204.stdout:756: set_chunk oid 4 target oid 2024-09-13T11:32:34.098 INFO:tasks.rados.rados.0.smithi204.stdout:756: set_chunk oid 4 offset: 602624 length: 879616 target oid 09da908872ab02fed20668d5b2250c35bd715c4347a1f161d33b8cf32e793a26 offset: 0 snap -1 2024-09-13T11:32:34.098 INFO:tasks.rados.rados.0.smithi204.stdout:757: write (excl) oid 10 current snap is 0 2024-09-13T11:32:34.098 INFO:tasks.rados.rados.0.smithi204.stdout:757: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-13T11:32:34.115 INFO:tasks.rados.rados.0.smithi204.stdout:757: writing smithi20438004-10 from 480700 to 1118189 tid 1 2024-09-13T11:32:34.115 INFO:tasks.rados.rados.0.smithi204.stdout:757: writing smithi20438004-10 from 1651964 to 2053649 tid 2 2024-09-13T11:32:34.118 INFO:tasks.rados.rados.0.smithi204.stdout:757: writing smithi20438004-10 from 2713358 to 3483971 tid 3 2024-09-13T11:32:34.118 INFO:tasks.rados.rados.0.smithi204.stdout:758: read oid 42 snap -1 2024-09-13T11:32:34.118 INFO:tasks.rados.rados.0.smithi204.stdout:758: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T11:32:34.118 INFO:tasks.rados.rados.0.smithi204.stdout:759: read oid 6 snap -1 2024-09-13T11:32:34.118 INFO:tasks.rados.rados.0.smithi204.stdout:759: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:34.119 INFO:tasks.rados.rados.0.smithi204.stdout:760: write (excl) oid 9 current snap is 0 2024-09-13T11:32:34.119 INFO:tasks.rados.rados.0.smithi204.stdout:748: finishing write tid 1 to smithi20438004-31 2024-09-13T11:32:34.119 INFO:tasks.rados.rados.0.smithi204.stdout:748: finishing write tid 2 to smithi20438004-31 2024-09-13T11:32:34.119 INFO:tasks.rados.rados.0.smithi204.stdout:760: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-13T11:32:34.124 INFO:tasks.rados.rados.0.smithi204.stdout:760: writing smithi20438004-9 from 497507 to 1210245 tid 1 2024-09-13T11:32:34.130 INFO:tasks.rados.rados.0.smithi204.stdout:760: writing smithi20438004-9 from 1710446 to 2372142 tid 2 2024-09-13T11:32:34.138 INFO:tasks.rados.rados.0.smithi204.stdout:760: writing smithi20438004-9 from 3140781 to 3697507 tid 3 2024-09-13T11:32:34.138 INFO:tasks.rados.rados.0.smithi204.stdout:761: tier_flush oid 43 2024-09-13T11:32:34.138 INFO:tasks.rados.rados.0.smithi204.stdout:761: tier_flush oid 43 snap -1 2024-09-13T11:32:34.138 INFO:tasks.rados.rados.0.smithi204.stdout:749: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-13T11:32:34.158 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1834836 2024-09-13T11:32:34.160 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.160 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.162 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:34.161+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:34.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:32:34.191 INFO:tasks.rados.rados.0.smithi204.stdout:748: finishing write tid 3 to smithi20438004-31 2024-09-13T11:32:34.191 INFO:tasks.rados.rados.0.smithi204.stdout:748: finishing write tid 4 to smithi20438004-31 2024-09-13T11:32:34.191 INFO:tasks.rados.rados.0.smithi204.stdout:748: finishing write tid 5 to smithi20438004-31 2024-09-13T11:32:34.191 INFO:tasks.rados.rados.0.smithi204.stdout:748: oid 31 updating version 0 to 25 2024-09-13T11:32:34.191 INFO:tasks.rados.rados.0.smithi204.stdout:748: oid 31 updating version 25 to 27 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:748: oid 31 version 27 is already newer than 26 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:748: oid 31 updating version 27 to 28 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 28 (ObjNum 95 snap 0 seq_num 95) dirty exists 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:748: left oid 31 (ObjNum 95 snap 0 seq_num 95) 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:750: finishing write tid 1 to smithi20438004-19 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:750: finishing write tid 2 to smithi20438004-19 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:750: finishing write tid 3 to smithi20438004-19 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:748: done (13 left) 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:749: done (12 left) 2024-09-13T11:32:34.192 INFO:tasks.rados.rados.0.smithi204.stdout:762: set_chunk oid 39 target oid 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:762: set_chunk oid 39 offset: 625152 length: 290304 target oid a2f2c330500f1b1d28c41e464817565848095218a89d32db0b731691bbe2b77b offset: 0 snap -1 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:763: read oid 48 snap -1 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:763: expect (ObjNum 79 snap 0 seq_num 79) 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:764: read oid 3 snap -1 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:764: expect deleted 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:765: write oid 5 current snap is 0 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:750: finishing write tid 4 to smithi20438004-19 2024-09-13T11:32:34.196 INFO:tasks.rados.rados.0.smithi204.stdout:750: finishing write tid 5 to smithi20438004-19 2024-09-13T11:32:34.197 INFO:tasks.rados.rados.0.smithi204.stdout:750: oid 19 updating version 0 to 54 2024-09-13T11:32:34.197 INFO:tasks.rados.rados.0.smithi204.stdout:750: oid 19 updating version 54 to 55 2024-09-13T11:32:34.197 INFO:tasks.rados.rados.0.smithi204.stdout:750: oid 19 version 55 is already newer than 52 2024-09-13T11:32:34.201 INFO:tasks.rados.rados.0.smithi204.stdout:750: oid 19 version 55 is already newer than 53 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 55 (ObjNum 96 snap 0 seq_num 96) dirty exists 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:750: left oid 19 (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: finishing write tid 1 to smithi20438004-22 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: finishing write tid 2 to smithi20438004-22 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:753: got (22) Invalid argument 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: finishing write tid 3 to smithi20438004-22 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: finishing write tid 4 to smithi20438004-22 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: finishing write tid 5 to smithi20438004-22 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: oid 22 updating version 0 to 38 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: oid 22 version 38 is already newer than 36 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: oid 22 version 38 is already newer than 35 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: oid 22 version 38 is already newer than 37 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 38 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-09-13T11:32:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:751: left oid 22 (ObjNum 97 snap 0 seq_num 97) 2024-09-13T11:32:34.203 INFO:tasks.rados.rados.0.smithi204.stdout:752: got (0) Success 2024-09-13T11:32:34.203 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-13T11:32:34.203 INFO:tasks.rados.rados.0.smithi204.stdout:765: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-13T11:32:34.203 INFO:tasks.rados.rados.0.smithi204.stdout:765: writing smithi20438004-5 from 514314 to 1302301 tid 1 2024-09-13T11:32:34.207 INFO:tasks.rados.rados.0.smithi204.stdout:765: writing smithi20438004-5 from 1768928 to 2174282 tid 2 2024-09-13T11:32:34.214 INFO:tasks.rados.rados.0.smithi204.stdout:765: writing smithi20438004-5 from 2768204 to 3277714 tid 3 2024-09-13T11:32:34.214 INFO:tasks.rados.rados.0.smithi204.stdout:750: done (15 left) 2024-09-13T11:32:34.214 INFO:tasks.rados.rados.0.smithi204.stdout:751: done (14 left) 2024-09-13T11:32:34.214 INFO:tasks.rados.rados.0.smithi204.stdout:752: done (13 left) 2024-09-13T11:32:34.214 INFO:tasks.rados.rados.0.smithi204.stdout:753: done (12 left) 2024-09-13T11:32:34.214 INFO:tasks.rados.rados.0.smithi204.stdout:766: read oid 15 snap -1 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:766: expect (ObjNum 67 snap 0 seq_num 67) 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:767: tier_promote oid 13 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: finishing write tid 5 to smithi20438004-17 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: oid 17 updating version 0 to 42 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: oid 17 version 42 is already newer than 41 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: oid 17 version 42 is already newer than 40 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: oid 17 version 42 is already newer than 39 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 42 (ObjNum 98 snap 0 seq_num 98) dirty exists 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:754: left oid 17 (ObjNum 98 snap 0 seq_num 98) 2024-09-13T11:32:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:755: finishing write tid 1 to smithi20438004-14 2024-09-13T11:32:34.216 INFO:tasks.rados.rados.0.smithi204.stdout:755: finishing write tid 2 to smithi20438004-14 2024-09-13T11:32:34.216 INFO:tasks.rados.rados.0.smithi204.stdout:754: done (13 left) 2024-09-13T11:32:34.216 INFO:tasks.rados.rados.0.smithi204.stdout:768: set_chunk oid 31 target oid 2024-09-13T11:32:34.242 INFO:tasks.rados.rados.0.smithi204.stdout:768: set_chunk oid 31 offset: 124928 length: 2923008 target oid 86e84ad357e4846a539d4df07011674aa0221b97892274eefe851fb6ba1029a5 offset: 0 snap -1 2024-09-13T11:32:34.242 INFO:tasks.rados.rados.0.smithi204.stdout:769: write oid 1 current snap is 0 2024-09-13T11:32:34.242 INFO:tasks.rados.rados.0.smithi204.stdout:755: finishing write tid 3 to smithi20438004-14 2024-09-13T11:32:34.243 INFO:tasks.rados.rados.0.smithi204.stdout:769: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-13T11:32:34.247 INFO:tasks.rados.rados.0.smithi204.stdout:769: writing smithi20438004-1 from 531121 to 994357 tid 1 2024-09-13T11:32:34.255 INFO:tasks.rados.rados.0.smithi204.stdout:769: writing smithi20438004-1 from 1711057 to 2376422 tid 2 2024-09-13T11:32:34.262 INFO:tasks.rados.rados.0.smithi204.stdout:769: writing smithi20438004-1 from 3079274 to 3731121 tid 3 2024-09-13T11:32:34.262 INFO:tasks.rados.rados.0.smithi204.stdout:770: set_chunk oid 8 target oid 2024-09-13T11:32:34.263 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:34.263+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:770: set_chunk oid 8 offset: 1458176 length: 325632 target oid 77a92c77d7459e1cc5adb693f309e31dfee8a90bac6c7913537b6086431505ab offset: 0 snap -1 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:755: finishing write tid 4 to smithi20438004-14 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:755: finishing write tid 5 to smithi20438004-14 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:755: oid 14 updating version 0 to 41 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:755: oid 14 updating version 41 to 42 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:755: oid 14 updating version 42 to 43 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:755: oid 14 updating version 43 to 44 2024-09-13T11:32:34.268 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 44 (ObjNum 99 snap 0 seq_num 99) dirty exists 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:755: left oid 14 (ObjNum 99 snap 0 seq_num 99) 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:756: finishing set_chunk to oid 4 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 602624 length: 879616 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:757: finishing write tid 1 to smithi20438004-10 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:757: finishing write tid 2 to smithi20438004-10 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:755: done (15 left) 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:756: done (14 left) 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:771: write (excl) oid 47 current snap is 0 2024-09-13T11:32:34.269 INFO:tasks.rados.rados.0.smithi204.stdout:758: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T11:32:34.277 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.277 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:32:34.332 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1003205 2024-09-13T11:32:34.350 INFO:tasks.rados.rados.0.smithi204.stdout:759: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:34.350 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:34.364 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:34.363+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:34.393 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1533027 2024-09-13T11:32:34.397 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.397 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.407 INFO:tasks.rados.rados.0.smithi204.stdout:757: finishing write tid 3 to smithi20438004-10 2024-09-13T11:32:34.407 INFO:tasks.rados.rados.0.smithi204.stdout:757: finishing write tid 4 to smithi20438004-10 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:757: finishing write tid 5 to smithi20438004-10 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:757: oid 10 updating version 0 to 62 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:757: oid 10 version 62 is already newer than 61 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:757: oid 10 version 62 is already newer than 60 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:757: oid 10 updating version 62 to 63 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 63 (ObjNum 100 snap 0 seq_num 100) dirty exists 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:757: left oid 10 (ObjNum 100 snap 0 seq_num 100) 2024-09-13T11:32:34.408 INFO:tasks.rados.rados.0.smithi204.stdout:760: finishing write tid 1 to smithi20438004-9 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: finishing write tid 2 to smithi20438004-9 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: finishing write tid 3 to smithi20438004-9 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: finishing write tid 4 to smithi20438004-9 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: finishing write tid 5 to smithi20438004-9 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: oid 9 updating version 0 to 29 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: oid 9 updating version 29 to 31 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: oid 9 updating version 31 to 32 2024-09-13T11:32:34.409 INFO:tasks.rados.rados.0.smithi204.stdout:760: oid 9 version 32 is already newer than 30 2024-09-13T11:32:34.410 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 32 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-13T11:32:34.410 INFO:tasks.rados.rados.0.smithi204.stdout:760: left oid 9 (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:34.410 INFO:tasks.rados.rados.0.smithi204.stdout:761: got (0) Success 2024-09-13T11:32:34.410 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-13T11:32:34.410 INFO:tasks.rados.rados.0.smithi204.stdout:771: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-13T11:32:34.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:32:34.412 INFO:tasks.rados.rados.0.smithi204.stdout:771: writing smithi20438004-47 from 547928 to 1086413 tid 1 2024-09-13T11:32:34.417 INFO:tasks.rados.rados.0.smithi204.stdout:771: writing smithi20438004-47 from 1769539 to 2178562 tid 2 2024-09-13T11:32:34.423 INFO:tasks.rados.rados.0.smithi204.stdout:771: writing smithi20438004-47 from 2706697 to 3238751 tid 3 2024-09-13T11:32:34.424 INFO:tasks.rados.rados.0.smithi204.stdout:757: done (14 left) 2024-09-13T11:32:34.424 INFO:tasks.rados.rados.0.smithi204.stdout:758: done (13 left) 2024-09-13T11:32:34.424 INFO:tasks.rados.rados.0.smithi204.stdout:759: done (12 left) 2024-09-13T11:32:34.424 INFO:tasks.rados.rados.0.smithi204.stdout:760: done (11 left) 2024-09-13T11:32:34.424 INFO:tasks.rados.rados.0.smithi204.stdout:761: done (10 left) 2024-09-13T11:32:34.424 INFO:tasks.rados.rados.0.smithi204.stdout:772: set_chunk oid 29 target oid 2024-09-13T11:32:34.431 INFO:tasks.rados.rados.0.smithi204.stdout:772: set_chunk oid 29 offset: 1354240 length: 812032 target oid 1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222 offset: 0 snap -1 2024-09-13T11:32:34.431 INFO:tasks.rados.rados.0.smithi204.stdout:773: set_chunk oid 45 target oid 2024-09-13T11:32:34.457 INFO:tasks.rados.rados.0.smithi204.stdout:773: set_chunk oid 45 offset: 83968 length: 2706432 target oid f0047526bc92329fbb756444431626dfe7d9487c03334a9a8b859c296c7c463d offset: 0 snap -1 2024-09-13T11:32:34.457 INFO:tasks.rados.rados.0.smithi204.stdout:774: write (excl) oid 11 current snap is 0 2024-09-13T11:32:34.457 INFO:tasks.rados.rados.0.smithi204.stdout:774: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-13T11:32:34.462 INFO:tasks.rados.rados.0.smithi204.stdout:774: writing smithi20438004-11 from 564735 to 1178469 tid 1 2024-09-13T11:32:34.466 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:34.464+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:34.470 INFO:tasks.rados.rados.0.smithi204.stdout:774: writing smithi20438004-11 from 1828021 to 2497055 tid 2 2024-09-13T11:32:34.477 INFO:tasks.rados.rados.0.smithi204.stdout:774: writing smithi20438004-11 from 3134120 to 3764735 tid 3 2024-09-13T11:32:34.477 INFO:tasks.rados.rados.0.smithi204.stdout:775: set_chunk oid 23 target oid 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:775: set_chunk oid 23 offset: 562688 length: 781824 target oid d3576626dc445f22e83e0484db33f4b15a69cce0c64f34f54a45b82f4a73709f offset: 0 snap -1 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:776: read oid 9 snap -1 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:776: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:777: read oid 30 snap -1 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:777: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:762: finishing set_chunk to oid 39 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 47 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-09-13T11:32:34.487 INFO:tasks.rados.rados.0.smithi204.stdout:763: expect (ObjNum 79 snap 0 seq_num 79) 2024-09-13T11:32:34.519 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.519 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:32:34.567 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:34.565+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:34.608 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1947431 2024-09-13T11:32:34.631 INFO:tasks.rados.rados.0.smithi204.stdout:765: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:765: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:762: done (15 left) 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:763: done (14 left) 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:764: done (13 left) 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:778: read oid 16 snap -1 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:778: expect deleted 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:779: read oid 39 snap -1 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:779: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:780: write (excl) oid 7 current snap is 0 2024-09-13T11:32:34.632 INFO:tasks.rados.rados.0.smithi204.stdout:765: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:34.633 INFO:tasks.rados.rados.0.smithi204.stdout:780: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-13T11:32:34.638 INFO:tasks.rados.rados.0.smithi204.stdout:780: writing smithi20438004-7 from 581542 to 1270525 tid 1 2024-09-13T11:32:34.644 INFO:tasks.rados.rados.0.smithi204.stdout:780: writing smithi20438004-7 from 1886503 to 2299195 tid 2 2024-09-13T11:32:34.650 INFO:tasks.rados.rados.0.smithi204.stdout:780: writing smithi20438004-7 from 2761543 to 3432494 tid 3 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:765: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:765: finishing write tid 5 to smithi20438004-5 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:765: oid 5 updating version 0 to 26 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:765: oid 5 updating version 26 to 27 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:765: oid 5 version 27 is already newer than 25 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:765: oid 5 updating version 27 to 28 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 28 (ObjNum 102 snap 0 seq_num 102) dirty exists 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:765: left oid 5 (ObjNum 102 snap 0 seq_num 102) 2024-09-13T11:32:34.651 INFO:tasks.rados.rados.0.smithi204.stdout:766: expect (ObjNum 67 snap 0 seq_num 67) 2024-09-13T11:32:34.657 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 766138 2024-09-13T11:32:34.667 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:34.666+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:34.695 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.695 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:32:34.707 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:34.768 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:34.767+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:34.811 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.811 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.820 INFO:tasks.rados.rados.0.smithi204.stdout:767: got (0) Success 2024-09-13T11:32:34.820 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-13T11:32:34.820 INFO:tasks.rados.rados.0.smithi204.stdout:768: finishing set_chunk to oid 31 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk returned error code -95 offset: 124928 length: 2923008 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 95 snap 0 seq_num 95) dirty exists 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:769: finishing write tid 1 to smithi20438004-1 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:769: finishing write tid 2 to smithi20438004-1 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:769: finishing write tid 3 to smithi20438004-1 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:765: done (15 left) 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:766: done (14 left) 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:767: done (13 left) 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:768: done (12 left) 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:781: read oid 40 snap -1 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:781: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:782: delete oid 36 current snap is 0 2024-09-13T11:32:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:769: finishing write tid 4 to smithi20438004-1 2024-09-13T11:32:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:769: finishing write tid 5 to smithi20438004-1 2024-09-13T11:32:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:769: oid 1 updating version 0 to 30 2024-09-13T11:32:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:769: oid 1 updating version 30 to 32 2024-09-13T11:32:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:769: oid 1 version 32 is already newer than 29 2024-09-13T11:32:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:769: oid 1 version 32 is already newer than 31 2024-09-13T11:32:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 32 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-09-13T11:32:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:769: left oid 1 (ObjNum 103 snap 0 seq_num 103) 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:770: finishing set_chunk to oid 8 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 1458176 length: 325632 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: finishing write tid 1 to smithi20438004-47 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: finishing write tid 2 to smithi20438004-47 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: finishing write tid 3 to smithi20438004-47 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: finishing write tid 4 to smithi20438004-47 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: finishing write tid 5 to smithi20438004-47 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: oid 47 updating version 0 to 22 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: oid 47 version 22 is already newer than 21 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: oid 47 updating version 22 to 23 2024-09-13T11:32:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:771: oid 47 version 23 is already newer than 20 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 23 (ObjNum 104 snap 0 seq_num 104) dirty exists 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:771: left oid 47 (ObjNum 104 snap 0 seq_num 104) 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:772: finishing set_chunk to oid 29 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 40 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:773: finishing set_chunk to oid 45 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 83968 length: 2706432 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:774: finishing write tid 1 to smithi20438004-11 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:774: finishing write tid 2 to smithi20438004-11 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:774: finishing write tid 3 to smithi20438004-11 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:774: finishing write tid 4 to smithi20438004-11 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:774: finishing write tid 5 to smithi20438004-11 2024-09-13T11:32:34.824 INFO:tasks.rados.rados.0.smithi204.stdout:774: oid 11 updating version 0 to 49 2024-09-13T11:32:34.825 INFO:tasks.rados.rados.0.smithi204.stdout:774: oid 11 version 49 is already newer than 46 2024-09-13T11:32:34.825 INFO:tasks.rados.rados.0.smithi204.stdout:774: oid 11 version 49 is already newer than 48 2024-09-13T11:32:34.825 INFO:tasks.rados.rados.0.smithi204.stdout:774: oid 11 version 49 is already newer than 47 2024-09-13T11:32:34.825 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 49 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-09-13T11:32:34.825 INFO:tasks.rados.rados.0.smithi204.stdout:774: left oid 11 (ObjNum 105 snap 0 seq_num 105) 2024-09-13T11:32:34.825 INFO:tasks.rados.rados.0.smithi204.stdout:776: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:34.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:32:34.869 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:34.868+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:34.878 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 211443 2024-09-13T11:32:34.970 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:34.970 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:34.971 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:34.970+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:34.980 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:32:35.010 INFO:tasks.rados.rados.0.smithi204.stdout:777: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-13T11:32:35.011 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:35.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:35.071 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:35.070+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:35.084 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:35.084 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:35.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:32:35.159 INFO:tasks.rados.rados.0.smithi204.stdout:775: finishing set_chunk to oid 23 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 28 (ObjNum 85 snap 0 seq_num 85) dirty exists 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:769: done (13 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:770: done (12 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:771: done (11 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:772: done (10 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:773: done (9 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:774: done (8 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:775: done (7 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:776: done (6 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:777: done (5 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:778: done (4 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:782: done (3 left) 2024-09-13T11:32:35.160 INFO:tasks.rados.rados.0.smithi204.stdout:783: read oid 5 snap -1 2024-09-13T11:32:35.161 INFO:tasks.rados.rados.0.smithi204.stdout:783: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-13T11:32:35.161 INFO:tasks.rados.rados.0.smithi204.stdout:784: read oid 20 snap -1 2024-09-13T11:32:35.161 INFO:tasks.rados.rados.0.smithi204.stdout:784: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:35.161 INFO:tasks.rados.rados.0.smithi204.stdout:779: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:35.161 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:35.161 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:35.172 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:35.170+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:35.177 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:35.178 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:35.178 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:35.194 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:35.195 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:35.211 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:35.211 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:35.229 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:35.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:35.238 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:35.238 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:35.239 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:35.245 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:35.245 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:35.247 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:32:35.261 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:35.262 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:35.272 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:35.271+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:35.277 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:35.278 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:35.291 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 680608 2024-09-13T11:32:35.294 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:35.294 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:35.311 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:35.312 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:35.329 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:35.329 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:35.345 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:35.345 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:35.349 INFO:tasks.rados.rados.0.smithi204.stdout:780: finishing write tid 1 to smithi20438004-7 2024-09-13T11:32:35.349 INFO:tasks.rados.rados.0.smithi204.stdout:780: finishing write tid 2 to smithi20438004-7 2024-09-13T11:32:35.349 INFO:tasks.rados.rados.0.smithi204.stdout:780: finishing write tid 3 to smithi20438004-7 2024-09-13T11:32:35.349 INFO:tasks.rados.rados.0.smithi204.stdout:780: finishing write tid 4 to smithi20438004-7 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:785: read oid 3 snap -1 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:785: expect deleted 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:779: done (5 left) 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:786: tier_evict oid 41 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:786: tier_evict oid 41 snap -1 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:780: finishing write tid 5 to smithi20438004-7 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:780: oid 7 updating version 0 to 28 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:780: oid 7 updating version 28 to 30 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:780: oid 7 version 30 is already newer than 27 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:780: oid 7 version 30 is already newer than 29 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 30 (ObjNum 106 snap 0 seq_num 106) dirty exists 2024-09-13T11:32:35.350 INFO:tasks.rados.rados.0.smithi204.stdout:780: left oid 7 (ObjNum 106 snap 0 seq_num 106) 2024-09-13T11:32:35.351 INFO:tasks.rados.rados.0.smithi204.stdout:781: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:35.361 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:35.362 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:35.373 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:35.372+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:35.378 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:35.378 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:35.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:35.377+0000 7f3f537fe640 0 -- 172.21.15.38:0/3482719992 >> v2:172.21.15.204:6812/501019151 conn(0x7f3f54220740 msgr2=0x7f3f54222b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'87, mlcod=24'81)", 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.319326716999999, 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0126724, 2024-09-13T11:32:35.393 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:35.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071634+0000", 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9599999999999996e-05 2024-09-13T11:32:35.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071640+0000", 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4980000000000003e-06 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071644+0000", 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2919999999999997e-06 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073407+0000", 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001762149 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073417+0000", 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0859999999999999e-05 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084190+0000", 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010772903 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084216+0000", 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5622e-05 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:118 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'88, mlcod=24'81)", 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.318806703, 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012784627999999999, 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4900000000000001e-07 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072067+0000", 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2260000000000001e-06 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072074+0000", 2024-09-13T11:32:35.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2820000000000002e-06 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072077+0000", 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5899999999999999e-06 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075773+0000", 2024-09-13T11:32:35.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003695884 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075786+0000", 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2627e-05 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084816+0000", 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090302000000000004 2024-09-13T11:32:35.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084848+0000", 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2369999999999997e-05 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:119 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'89, mlcod=24'81)", 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.318356649999998, 2024-09-13T11:32:35.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012362191, 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072514+0000", 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072521+0000", 2024-09-13T11:32:35.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6540000000000002e-06 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072531+0000", 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8309999999999996e-06 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072536+0000", 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3090000000000003e-06 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076045+0000", 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035092769999999999 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076058+0000", 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2992e-05 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084850+0000", 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087915289999999993 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084876+0000", 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5947000000000001e-05 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:120 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'90, mlcod=24'81)", 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072941+0000", 2024-09-13T11:32:35.409 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.317928969, 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011957627, 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072941+0000", 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072941+0000", 2024-09-13T11:32:35.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072942+0000", 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.32e-07 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073044+0000", 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010217099999999999 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073054+0000", 2024-09-13T11:32:35.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7899999999999994e-06 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073058+0000", 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076266+0000", 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003207938 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.076278+0000", 2024-09-13T11:32:35.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1464000000000001e-05 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084877+0000", 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085992389999999998 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084899+0000", 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1798e-05 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:121 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'91, mlcod=24'81)", 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.073396+0000", 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.317473812999999, 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011525894, 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073396+0000", 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073396+0000", 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073397+0000", 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.68e-07 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073404+0000", 2024-09-13T11:32:35.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5599999999999999e-06 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073410+0000", 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7900000000000002e-06 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073415+0000", 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5920000000000002e-06 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081268+0000", 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078531829999999997 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081280+0000", 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2065999999999999e-05 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084901+0000", 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036210040000000001 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084922+0000", 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1131000000000001e-05 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:122 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'92, mlcod=24'82)", 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.316853956999999, 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013166437, 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.418 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074017+0000", 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9500000000000003e-07 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074022+0000", 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0529999999999997e-06 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074029+0000", 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8419999999999999e-06 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074032+0000", 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8060000000000001e-06 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081498+0000", 2024-09-13T11:32:35.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074659669999999996 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081510+0000", 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1265e-05 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087152+0000", 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056421889999999997 2024-09-13T11:32:35.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087183+0000", 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.082e-05 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'82)", 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:35.423 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.316341864000002, 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012918696, 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074529+0000", 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8400000000000004e-07 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074601+0000", 2024-09-13T11:32:35.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2106000000000004e-05 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074611+0000", 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0054000000000001e-05 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074616+0000", 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6349999999999997e-06 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084250+0000", 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096340109999999996 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084263+0000", 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3292000000000001e-05 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087416+0000", 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003152672 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087447+0000", 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1241999999999997e-05 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'82)", 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.315816672, 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012417658, 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.428 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5600000000000001e-07 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075058+0000", 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6689999999999999e-06 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075065+0000", 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9269999999999999e-06 2024-09-13T11:32:35.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075069+0000", 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.963e-06 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084645+0000", 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095762209999999993 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084657+0000", 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2017e-05 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087449+0000", 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002792189 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087471+0000", 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2215999999999999e-05 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.036558210999999, 2024-09-13T11:32:35.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:32:35.434 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.434 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:32:35.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:32:35.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:209 3.2 e38/30 3:6afe79ef:::smithi20438004-10 low_tier:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2980272030000002, 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016793596000000001, 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4799999999999999e-07 2024-09-13T11:32:35.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102771+0000", 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0099271120000000001 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102791+0000", 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0148000000000001e-05 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102801+0000", 2024-09-13T11:32:35.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8560000000000002e-06 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102898+0000", 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7528000000000005e-05 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102915+0000", 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6915999999999999e-05 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109590+0000", 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066751579999999996 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109637+0000", 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6529999999999997e-05 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:35.446 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2880272900000005, 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:32:35.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2758096959999996, 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:35.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:32:35.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:207 3.2 e38/30 3:59a396eb:::smithi20438004-8 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:32:35.455 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2690599920000007, 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025976583000000001, 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121811+0000", 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2799999999999996e-07 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128273+0000", 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064622129999999996 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128294+0000", 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1364000000000001e-05 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128312+0000", 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8002000000000001e-05 2024-09-13T11:32:35.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128443+0000", 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013052699999999999 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128487+0000", 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4338999999999997e-05 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147644+0000", 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019156341 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147787+0000", 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014326900000000001 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.460 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:220 3.2 e38/30 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2553923620000003, 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013667876000000001, 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.461 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135479+0000", 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.175e-06 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139134+0000", 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036550240000000002 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139152+0000", 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8136999999999999e-05 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139168+0000", 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5951e-05 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139254+0000", 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5691000000000003e-05 2024-09-13T11:32:35.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139278+0000", 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4148e-05 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149117+0000", 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098384820000000008 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149146+0000", 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9268000000000002e-05 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:233 3.2 e38/30 3:53d0adee:::smithi20438004-34 low_tier:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2379118429999991, 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023237916000000001, 2024-09-13T11:32:35.466 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:35.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152960+0000", 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161956+0000", 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0089955509999999992 2024-09-13T11:32:35.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161973+0000", 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7350000000000002e-05 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161984+0000", 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.11e-05 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162047+0000", 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2677999999999999e-05 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162075+0000", 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8501e-05 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176139+0000", 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014063512 2024-09-13T11:32:35.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176196+0000", 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7661000000000002e-05 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:240 3.2 e38/30 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:35.471 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2248721180000004, 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028057044, 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:35.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166000+0000", 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179000+0000", 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013000523999999999 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179014+0000", 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4143e-05 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179023+0000", 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5969999999999992e-06 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179083+0000", 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9911000000000003e-05 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179110+0000", 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7251999999999999e-05 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014884616999999999 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.194055+0000", 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.475 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:246 3.2 e38/30 3:68241bcb:::smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'6)", 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.2033604889999996, 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013298119000000001, 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:35.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187511+0000", 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000003e-07 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197044+0000", 2024-09-13T11:32:35.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095328879999999998 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197061+0000", 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7413000000000001e-05 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197072+0000", 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1078e-05 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197146+0000", 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4449999999999994e-05 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197176+0000", 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9584e-05 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.200753+0000", 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035767020000000002 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.200808+0000", 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5120000000000001e-05 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:244 3.2 e38/30 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'6)", 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.1937595779999999, 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023733269000000001, 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.480 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204547+0000", 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074358269999999999 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204567+0000", 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0196000000000001e-05 2024-09-13T11:32:35.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204579+0000", 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2071e-05 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204679+0000", 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9702000000000002e-05 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204727+0000", 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7929000000000001e-05 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220809+0000", 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016082012999999999 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220844+0000", 2024-09-13T11:32:35.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4959999999999997e-05 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:248 3.2 e38/30 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'11, mlcod=38'8)", 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.1862583700000009, 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016257252999999999, 2024-09-13T11:32:35.485 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204613+0000", 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5300000000000003e-07 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210186+0000", 2024-09-13T11:32:35.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005573789 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210202+0000", 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5424999999999999e-05 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210215+0000", 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3393e-05 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210285+0000", 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9295000000000007e-05 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210305+0000", 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0557999999999999e-05 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220848+0000", 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010542463 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220869+0000", 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1577e-05 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.490 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:247 3.2 e38/30 3:4f19c0b8:::smithi20438004-48 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.1806272250000003, 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016148066999999999, 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210244+0000", 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4099999999999997e-07 2024-09-13T11:32:35.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217679+0000", 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074353470000000001 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217694+0000", 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.455e-05 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217702+0000", 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-13T11:32:35.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217760+0000", 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7788000000000003e-05 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217786+0000", 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5803999999999999e-05 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226348+0000", 2024-09-13T11:32:35.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085624180000000005 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226391+0000", 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2660999999999997e-05 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.495 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:35.497 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:35.497 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:35.495+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:35.499 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:32:35.499 INFO:tasks.rados.rados.0.smithi204.stdout:784: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:35.499 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 271289 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:get_pg_num_history pg_num_history pg_num_history(e39 pg_nums {1={23=1},2={25=8},3={30=4},4={31=16}} deleted_pools ) 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:Importing pgid 4.4 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:Write #4:2451796c:::smithi20438004-17:head# 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:snapset 0=[]:{} 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:Write #4:28ee04de:::smithi20438004-15:head# 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:snapset 0=[]:{} 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:Write #4:2b094039:::smithi20438004-40:head# 2024-09-13T11:32:35.500 INFO:teuthology.orchestra.run.smithi154.stdout:snapset 0=[]:{} 2024-09-13T11:32:35.501 INFO:teuthology.orchestra.run.smithi154.stdout:write_pg epoch 39 info 4.4( v 39'34 (39'31,39'34] local-lis/les=31/32 n=3 ec=31/31 lis/c=31/31 les/c/f=32/32/0 sis=31) 2024-09-13T11:32:35.501 INFO:teuthology.orchestra.run.smithi154.stdout:Import successful 2024-09-13T11:32:35.511 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:35.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:32:35.561 INFO:tasks.rados.rados.0.smithi204.stdout:783: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-13T11:32:35.561 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:35.598 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:35.597+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:35.612 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:35.613 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:35.615 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:35.615 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:35.615 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:35.616 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:35.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:35.624 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:32:35.698 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:35.697+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:35.728 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:35.728 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:35.729 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:35.729 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:35.729 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:35.737 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:786: got (22) Invalid argument 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:780: done (5 left) 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:781: done (4 left) 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:783: done (3 left) 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:784: done (2 left) 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:785: done (1 left) 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:786: done (0 left) 2024-09-13T11:32:35.768 INFO:tasks.rados.rados.0.smithi204.stdout:787: delete oid 15 current snap is 0 2024-09-13T11:32:35.770 INFO:tasks.rados.rados.0.smithi204.stdout:787: done (0 left) 2024-09-13T11:32:35.771 INFO:tasks.rados.rados.0.smithi204.stdout:788: tier_promote oid 20 2024-09-13T11:32:35.771 INFO:tasks.rados.rados.0.smithi204.stdout:789: read oid 43 snap -1 2024-09-13T11:32:35.771 INFO:tasks.rados.rados.0.smithi204.stdout:789: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:35.771 INFO:tasks.rados.rados.0.smithi204.stdout:790: write oid 23 current snap is 0 2024-09-13T11:32:35.771 INFO:tasks.rados.rados.0.smithi204.stdout:790: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-09-13T11:32:35.777 INFO:tasks.rados.rados.0.smithi204.stdout:790: writing smithi20438004-23 from 631963 to 1263046 tid 1 2024-09-13T11:32:35.785 INFO:tasks.rados.rados.0.smithi204.stdout:790: writing smithi20438004-23 from 2061949 to 2621968 tid 2 2024-09-13T11:32:35.791 INFO:tasks.rados.rados.0.smithi204.stdout:790: writing smithi20438004-23 from 3243812 to 3748579 tid 3 2024-09-13T11:32:35.792 INFO:tasks.rados.rados.0.smithi204.stdout:791: tier_flush oid 12 2024-09-13T11:32:35.792 INFO:tasks.rados.rados.0.smithi204.stdout:791: tier_flush oid 12 snap -1 2024-09-13T11:32:35.792 INFO:tasks.rados.rados.0.smithi204.stdout:789: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:35.799 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:35.798+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:35.810 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1822585 2024-09-13T11:32:35.872 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1471526 2024-09-13T11:32:35.879 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:35.879 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:35.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:35.879 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:35.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.879 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:110 4.0 e38/31 4:001d90c2:::smithi20438004-28:head v 38'5, mlcod=38'3)", 2024-09-13T11:32:35.879 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:26.787247+0000", 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.0898164900000005, 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040545759999999998, 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.787247+0000", 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.787247+0000", 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.787248+0000", 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7299999999999996e-07 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.787975+0000", 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00072690799999999996 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.787984+0000", 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8179999999999993e-06 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.787990+0000", 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9730000000000002e-06 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.788028+0000", 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8117000000000001e-05 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.788040+0000", 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1864e-05 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.791274+0000", 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032338330000000002 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.791302+0000", 2024-09-13T11:32:35.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.849e-05 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.7798918730000004, 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:32:35.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:32:35.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:32:35.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:32:35.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:32:35.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:35.889 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.7674577280000001, 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:32:35.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.7575041339999995, 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.894 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.898 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:224 3.3 e38/30 3:e7e3acce:::smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=38'1)", 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.7328648879999999, 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010654663, 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144201+0000", 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1969999999999999e-06 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150234+0000", 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060323859999999998 2024-09-13T11:32:35.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150248+0000", 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4584e-05 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150256+0000", 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1270000000000003e-06 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150306+0000", 2024-09-13T11:32:35.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9633999999999999e-05 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150327+0000", 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0579999999999999e-05 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154794+0000", 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004467108 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154854+0000", 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0047000000000003e-05 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.7267863269999992, 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.903 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:32:35.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:32:35.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:35.907 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.7079630080000001, 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:35.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6944436590000009, 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:32:35.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6848033260000008, 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:32:35.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:32:35.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:32:35.921 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6744978259999996, 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:32:35.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:32:35.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6641118840000004, 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.926 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:32:35.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:32:35.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.3.22:1 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6553451819999996, 2024-09-13T11:32:35.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048600099999999997, 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221724+0000", 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7090000000000001e-06 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221732+0000", 2024-09-13T11:32:35.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1599999999999998e-06 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221737+0000", 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3939999999999998e-06 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221777+0000", 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0666e-05 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221789+0000", 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1986e-05 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226541+0000", 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047513249999999998 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226579+0000", 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8158999999999998e-05 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.22:3 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.222270+0000", 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6547939790000008, 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043380340000000002, 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222270+0000", 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222270+0000", 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222271+0000", 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222296+0000", 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4929000000000001e-05 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222304+0000", 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3729999999999993e-06 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222309+0000", 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1089999999999997e-06 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222348+0000", 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8543e-05 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222362+0000", 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4616e-05 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226581+0000", 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042188269999999996 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226608+0000", 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7046000000000002e-05 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.9.22:4 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'16, mlcod=38'12)", 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.232331+0000", 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.644732587, 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040868809999999997, 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.939 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232331+0000", 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232331+0000", 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232332+0000", 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8199999999999999e-07 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232337+0000", 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6519999999999997e-06 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232349+0000", 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2138e-05 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232355+0000", 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7230000000000001e-06 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232412+0000", 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7546000000000001e-05 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232434+0000", 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1617999999999999e-05 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.236384+0000", 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003950149 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.236418+0000", 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4372999999999998e-05 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.22:13 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'19, mlcod=38'17)", 2024-09-13T11:32:35.943 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.240816+0000", 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6362475090000004, 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0043295190000000004, 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240816+0000", 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240816+0000", 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240817+0000", 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240830+0000", 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2431e-05 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240840+0000", 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.064e-05 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240848+0000", 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8390000000000007e-06 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240882+0000", 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3958999999999998e-05 2024-09-13T11:32:35.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240895+0000", 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3022e-05 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.245112+0000", 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042165529999999996 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.245146+0000", 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4251000000000001e-05 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.12.22:18 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'22, mlcod=38'19)", 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.253030+0000", 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6240335600000009, 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051860980000000001, 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.948 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253030+0000", 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253030+0000", 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253031+0000", 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3300000000000002e-07 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253036+0000", 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2480000000000004e-06 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253045+0000", 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9719999999999998e-06 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253052+0000", 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.409e-06 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253113+0000", 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1438000000000004e-05 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253133+0000", 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9823e-05 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.258182+0000", 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050486339999999998 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.258217+0000", 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4941000000000003e-05 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.952 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.9.22:9 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'24, mlcod=38'21)", 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6096981929999998, 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049262630000000002, 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3099999999999999e-07 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267370+0000", 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1740000000000002e-06 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267376+0000", 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1750000000000002e-06 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267381+0000", 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9930000000000003e-06 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267656+0000", 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027412500000000001 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267676+0000", 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0488000000000001e-05 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272254+0000", 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045782499999999999 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272292+0000", 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7727e-05 2024-09-13T11:32:35.956 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.12.22:19 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'25, mlcod=38'21)", 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.267885+0000", 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.6091792280000004, 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044361610000000001, 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267885+0000", 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267885+0000", 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267886+0000", 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267893+0000", 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8269999999999998e-06 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267906+0000", 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3152e-05 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267912+0000", 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9880000000000001e-06 2024-09-13T11:32:35.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267993+0000", 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0130999999999996e-05 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.268020+0000", 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6984000000000001e-05 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272295+0000", 2024-09-13T11:32:35.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004275203 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272321+0000", 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6095999999999999e-05 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:772 3.3 e38/30 3:dd3ad6bf:::4ec2dd001b7153a0bb00c76a8320dec150c3eb87238ccdd9135be5e948aee2d9:head v 38'30, mlcod=38'28)", 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.359518+0000", 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.517546329, 2024-09-13T11:32:35.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068126380000000002, 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.359518+0000", 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.359518+0000", 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.359519+0000", 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5659999999999999e-06 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362100+0000", 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025806969999999999 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362119+0000", 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9320000000000001e-05 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362131+0000", 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1438e-05 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362189+0000", 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8730999999999997e-05 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362207+0000", 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7655999999999998e-05 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.366279+0000", 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040718969999999997 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.366330+0000", 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1332999999999998e-05 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.965 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:822 3.3 e39/30 3:c8f80c84:::bc036ea868a6b2c601148e4e87101fdc0cdd45a73131aa4e7533f4932f45b24d:head v 39'36, mlcod=39'34)", 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.879474+0000", 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 7.9975897419999997, 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004071674, 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.879474+0000", 2024-09-13T11:32:35.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.879474+0000", 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.879477+0000", 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7800000000000001e-06 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:35.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.879952+0000", 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00047505099999999999 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.879972+0000", 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9718e-05 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.879981+0000", 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6630000000000005e-06 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:35.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.880039+0000", 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7170999999999999e-05 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.880058+0000", 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9772e-05 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.883516+0000", 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003457267 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:35.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.883546+0000", 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0252000000000001e-05 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:35.970 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:35.972 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:32:35.972 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2135978 2024-09-13T11:32:35.972 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:35.970+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:35.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /var/log/ceph/exp.4.4.2 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:788: got (0) Success 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:790: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:790: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:788: done (3 left) 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:789: done (2 left) 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:792: tier_promote oid 39 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:790: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:35.997 INFO:tasks.rados.rados.0.smithi204.stdout:790: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:790: finishing write tid 5 to smithi20438004-23 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:790: oid 23 updating version 0 to 41 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:790: oid 23 version 41 is already newer than 38 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:790: oid 23 version 41 is already newer than 39 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:790: oid 23 version 41 is already newer than 40 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 41 (ObjNum 109 snap 0 seq_num 109) dirty exists 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:790: left oid 23 (ObjNum 109 snap 0 seq_num 109) 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:791: got (0) Success 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 86 snap 0 seq_num 86) dirty exists 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:790: done (2 left) 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:791: done (1 left) 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:793: write (excl) oid 23 current snap is 0 2024-09-13T11:32:35.998 INFO:tasks.rados.rados.0.smithi204.stdout:793: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-13T11:32:36.003 INFO:tasks.rados.rados.0.smithi204.stdout:793: writing smithi20438004-23 from 648770 to 1355102 tid 1 2024-09-13T11:32:36.011 INFO:tasks.rados.rados.0.smithi204.stdout:793: writing smithi20438004-23 from 2120431 to 2540461 tid 2 2024-09-13T11:32:36.017 INFO:tasks.rados.rados.0.smithi204.stdout:793: writing smithi20438004-23 from 3271235 to 3703627 tid 3 2024-09-13T11:32:36.017 INFO:tasks.rados.rados.0.smithi204.stdout:794: write (excl) oid 21 current snap is 0 2024-09-13T11:32:36.017 INFO:tasks.rados.rados.0.smithi204.stdout:794: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-13T11:32:36.025 INFO:tasks.rados.rados.0.smithi204.stdout:794: writing smithi20438004-21 from 665577 to 1447158 tid 1 2024-09-13T11:32:36.031 INFO:tasks.rados.rados.0.smithi204.stdout:794: writing smithi20438004-21 from 2062560 to 2626248 tid 2 2024-09-13T11:32:36.038 INFO:tasks.rados.rados.0.smithi204.stdout:794: writing smithi20438004-21 from 3182305 to 3825969 tid 3 2024-09-13T11:32:36.038 INFO:tasks.rados.rados.0.smithi204.stdout:795: write oid 14 current snap is 0 2024-09-13T11:32:36.038 INFO:tasks.rados.rados.0.smithi204.stdout:792: got (0) Success 2024-09-13T11:32:36.038 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 47 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-09-13T11:32:36.039 INFO:tasks.rados.rados.0.smithi204.stdout:793: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:36.039 INFO:tasks.rados.rados.0.smithi204.stdout:795: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-13T11:32:36.044 INFO:tasks.rados.rados.0.smithi204.stdout:795: writing smithi20438004-14 from 682384 to 1139214 tid 1 2024-09-13T11:32:36.053 INFO:tasks.rados.rados.0.smithi204.stdout:795: writing smithi20438004-14 from 1721042 to 2144741 tid 2 2024-09-13T11:32:36.056 INFO:tasks.rados.rados.0.smithi204.stdout:795: writing smithi20438004-14 from 2809728 to 3264664 tid 3 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:795: writing smithi20438004-14 from 3731357 to 3882384 tid 4 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:792: done (3 left) 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:796: write oid 35 current snap is 0 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:793: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:793: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:793: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:793: finishing write tid 5 to smithi20438004-23 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:793: oid 23 updating version 0 to 43 2024-09-13T11:32:36.060 INFO:tasks.rados.rados.0.smithi204.stdout:793: oid 23 updating version 43 to 45 2024-09-13T11:32:36.061 INFO:tasks.rados.rados.0.smithi204.stdout:793: oid 23 version 45 is already newer than 44 2024-09-13T11:32:36.061 INFO:tasks.rados.rados.0.smithi204.stdout:793: oid 23 version 45 is already newer than 42 2024-09-13T11:32:36.061 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 45 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-09-13T11:32:36.061 INFO:tasks.rados.rados.0.smithi204.stdout:793: left oid 23 (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:36.061 INFO:tasks.rados.rados.0.smithi204.stdout:794: finishing write tid 1 to smithi20438004-21 2024-09-13T11:32:36.061 INFO:tasks.rados.rados.0.smithi204.stdout:796: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-13T11:32:36.066 INFO:tasks.rados.rados.0.smithi204.stdout:796: writing smithi20438004-35 from 699191 to 1231270 tid 1 2024-09-13T11:32:36.073 INFO:tasks.rados.rados.0.smithi204.stdout:796: writing smithi20438004-35 from 1779524 to 2346881 tid 2 2024-09-13T11:32:36.080 INFO:tasks.rados.rados.0.smithi204.stdout:796: writing smithi20438004-35 from 2837151 to 3503359 tid 3 2024-09-13T11:32:36.080 INFO:tasks.rados.rados.0.smithi204.stdout:793: done (3 left) 2024-09-13T11:32:36.080 INFO:tasks.rados.rados.0.smithi204.stdout:797: tier_promote oid 41 2024-09-13T11:32:36.080 INFO:tasks.rados.rados.0.smithi204.stdout:794: finishing write tid 2 to smithi20438004-21 2024-09-13T11:32:36.080 INFO:tasks.rados.rados.0.smithi204.stdout:794: finishing write tid 3 to smithi20438004-21 2024-09-13T11:32:36.080 INFO:tasks.rados.rados.0.smithi204.stdout:794: finishing write tid 4 to smithi20438004-21 2024-09-13T11:32:36.081 INFO:tasks.rados.rados.0.smithi204.stdout:798: read oid 25 snap -1 2024-09-13T11:32:36.081 INFO:tasks.rados.rados.0.smithi204.stdout:798: expect (ObjNum 59 snap 0 seq_num 59) 2024-09-13T11:32:36.081 INFO:tasks.rados.rados.0.smithi204.stdout:799: read oid 40 snap -1 2024-09-13T11:32:36.081 INFO:tasks.rados.rados.0.smithi204.stdout:799: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:36.081 INFO:tasks.rados.rados.0.smithi204.stdout:800: set_chunk oid 6 target oid 2024-09-13T11:32:36.083 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:36.082+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:36.095 INFO:tasks.rados.rados.0.smithi204.stdout:800: set_chunk oid 6 offset: 184832 length: 1144832 target oid d07c99508293d84e4ef6e64f0dc5690227be6f0a554acee36ad011709b311285 offset: 0 snap -1 2024-09-13T11:32:36.095 INFO:tasks.rados.rados.0.smithi204.stdout:801: read oid 34 snap -1 2024-09-13T11:32:36.095 INFO:tasks.rados.rados.0.smithi204.stdout:801: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T11:32:36.095 INFO:tasks.rados.rados.0.smithi204.stdout:802: read oid 2 snap -1 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:802: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:794: finishing write tid 5 to smithi20438004-21 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:794: oid 21 updating version 0 to 11 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:794: oid 21 updating version 11 to 14 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:794: oid 21 version 14 is already newer than 13 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:794: oid 21 version 14 is already newer than 12 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 14 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:794: left oid 21 (ObjNum 111 snap 0 seq_num 111) 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:803: read oid 42 snap -1 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:803: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:794: done (9 left) 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:804: tier_flush oid 49 2024-09-13T11:32:36.096 INFO:tasks.rados.rados.0.smithi204.stdout:804: tier_flush oid 49 snap -1 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:795: finishing write tid 1 to smithi20438004-14 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:795: finishing write tid 2 to smithi20438004-14 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:805: tier_evict oid 30 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:805: tier_evict oid 30 snap -1 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:795: finishing write tid 3 to smithi20438004-14 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:795: finishing write tid 4 to smithi20438004-14 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:795: finishing write tid 5 to smithi20438004-14 2024-09-13T11:32:36.097 INFO:tasks.rados.rados.0.smithi204.stdout:806: read oid 3 snap -1 2024-09-13T11:32:36.098 INFO:tasks.rados.rados.0.smithi204.stdout:806: expect deleted 2024-09-13T11:32:36.098 INFO:tasks.rados.rados.0.smithi204.stdout:807: read oid 8 snap -1 2024-09-13T11:32:36.098 INFO:tasks.rados.rados.0.smithi204.stdout:807: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:36.098 INFO:tasks.rados.rados.0.smithi204.stdout:808: read oid 20 snap -1 2024-09-13T11:32:36.098 INFO:tasks.rados.rados.0.smithi204.stdout:808: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:36.098 INFO:tasks.rados.rados.0.smithi204.stdout:809: write (excl) oid 37 current snap is 0 2024-09-13T11:32:36.098 INFO:tasks.rados.rados.0.smithi204.stdout:809: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-13T11:32:36.102 INFO:tasks.rados.rados.0.smithi204.stdout:809: writing smithi20438004-37 from 715998 to 1323326 tid 1 2024-09-13T11:32:36.108 INFO:tasks.rados.rados.0.smithi204.stdout:809: writing smithi20438004-37 from 1838006 to 2265374 tid 2 2024-09-13T11:32:36.115 INFO:tasks.rados.rados.0.smithi204.stdout:809: writing smithi20438004-37 from 2864574 to 3342054 tid 3 2024-09-13T11:32:36.115 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:36.115 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:36.116 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:36.116 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:809: writing smithi20438004-37 from 3863835 to 3915998 tid 4 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:810: write (excl) oid 17 current snap is 0 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:795: finishing write tid 6 to smithi20438004-14 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:795: oid 14 updating version 0 to 54 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:795: oid 14 version 54 is already newer than 52 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:795: oid 14 version 54 is already newer than 50 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:795: oid 14 version 54 is already newer than 51 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:795: oid 14 version 54 is already newer than 53 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 54 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:795: left oid 14 (ObjNum 112 snap 0 seq_num 112) 2024-09-13T11:32:36.119 INFO:tasks.rados.rados.0.smithi204.stdout:796: finishing write tid 1 to smithi20438004-35 2024-09-13T11:32:36.120 INFO:tasks.rados.rados.0.smithi204.stdout:796: finishing write tid 2 to smithi20438004-35 2024-09-13T11:32:36.120 INFO:tasks.rados.rados.0.smithi204.stdout:810: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-13T11:32:36.127 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:32:36.127 INFO:tasks.rados.rados.0.smithi204.stdout:810: writing smithi20438004-17 from 732805 to 1531735 tid 1 2024-09-13T11:32:36.134 INFO:tasks.rados.rados.0.smithi204.stdout:810: writing smithi20438004-17 from 2296488 to 2867514 tid 2 2024-09-13T11:32:36.139 INFO:tasks.rados.rados.0.smithi204.stdout:810: writing smithi20438004-17 from 3291997 to 3697102 tid 3 2024-09-13T11:32:36.139 INFO:tasks.rados.rados.0.smithi204.stdout:795: done (15 left) 2024-09-13T11:32:36.139 INFO:tasks.rados.rados.0.smithi204.stdout:811: read oid 33 snap -1 2024-09-13T11:32:36.140 INFO:tasks.rados.rados.0.smithi204.stdout:811: expect deleted 2024-09-13T11:32:36.140 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:36.140 INFO:tasks.rados.rados.0.smithi204.stdout:799: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-13T11:32:36.173 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:36.172+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:36.238 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:36.238 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:36.238 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:36.239 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:36.239 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:36.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:32:36.274 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:36.273+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:36.292 INFO:tasks.rados.rados.0.smithi204.stdout:796: finishing write tid 3 to smithi20438004-35 2024-09-13T11:32:36.292 INFO:tasks.rados.rados.0.smithi204.stdout:796: finishing write tid 4 to smithi20438004-35 2024-09-13T11:32:36.292 INFO:tasks.rados.rados.0.smithi204.stdout:799: done (15 left) 2024-09-13T11:32:36.292 INFO:tasks.rados.rados.0.smithi204.stdout:812: read oid 9 snap -1 2024-09-13T11:32:36.292 INFO:tasks.rados.rados.0.smithi204.stdout:812: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:796: finishing write tid 5 to smithi20438004-35 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:796: oid 35 updating version 0 to 65 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:796: oid 35 updating version 65 to 66 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:796: oid 35 updating version 66 to 67 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:796: oid 35 updating version 67 to 68 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 68 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:796: left oid 35 (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:797: got (0) Success 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 39 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:798: expect (ObjNum 59 snap 0 seq_num 59) 2024-09-13T11:32:36.293 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:36.375 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:36.374+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:36.387 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1959292 2024-09-13T11:32:36.393 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:36.393 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:36.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:36.393 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:125 4.a 4:56ef4fd4:::smithi20438004-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:26.828738+0000", 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5625878899999996, 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011003384, 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 125 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.394 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.828738+0000", 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.828738+0000", 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.828739+0000", 2024-09-13T11:32:36.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.66e-06 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.829129+0000", 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038942000000000002 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.829140+0000", 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0815000000000001e-05 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.829145+0000", 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.04e-06 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.829232+0000", 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7303000000000001e-05 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.829541+0000", 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00030939500000000002 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.829814+0000", 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000272785 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.833051+0000", 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032372030000000001 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.833507+0000", 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045535299999999998 2024-09-13T11:32:36.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.839638+0000", 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061308170000000002 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.839690+0000", 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1946e-05 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.839741+0000", 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1647000000000002e-05 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:206 3.1 e38/30 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.092280+0000", 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.29904552, 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078188330000000007, 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.400 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092280+0000", 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092280+0000", 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092280+0000", 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.36e-07 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.095424+0000", 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031437539999999999 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.095435+0000", 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1100999999999999e-05 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.095443+0000", 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8849999999999999e-06 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.095503+0000", 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0123e-05 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.095523+0000", 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9721e-05 2024-09-13T11:32:36.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.100047+0000", 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045236039999999996 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.100099+0000", 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2408999999999999e-05 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:201 3.1 e38/30 3:b265d015:::smithi20438004-2 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097913+0000", 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2934126379999995, 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095082380000000005, 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097913+0000", 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097913+0000", 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097914+0000", 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4e-07 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.100877+0000", 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002962868 2024-09-13T11:32:36.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.100888+0000", 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1491999999999999e-05 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.100897+0000", 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1409999999999994e-06 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.100954+0000", 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6252000000000003e-05 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.100975+0000", 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1044999999999999e-05 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.107380+0000", 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064056800000000004 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.107421+0000", 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0920000000000001e-05 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2902471559999995, 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016706373, 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.409 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.99e-07 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104537+0000", 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034575700000000001 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104552+0000", 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5475999999999999e-05 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104566+0000", 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3766e-05 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104623+0000", 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7046999999999998e-05 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104650+0000", 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6786000000000001e-05 2024-09-13T11:32:36.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117752+0000", 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013102387 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117785+0000", 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2741999999999999e-05 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2862878299999991, 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025912691000000002, 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105039+0000", 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7100000000000001e-07 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108050+0000", 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030115910000000001 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108061+0000", 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0512e-05 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108069+0000", 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4479999999999994e-06 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108126+0000", 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7024999999999997e-05 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108165+0000", 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8701000000000001e-05 2024-09-13T11:32:36.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130916+0000", 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022750829 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130951+0000", 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4913999999999998e-05 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.418 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:203 3.1 e38/30 3:970ba768:::smithi20438004-4 low_tier:head v 38'5, mlcod=0'0)", 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2830813370000005, 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031827052000000002, 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108245+0000", 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1800000000000005e-07 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111202+0000", 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029571440000000001 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111221+0000", 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8887000000000001e-05 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111248+0000", 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6458e-05 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111707+0000", 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045935799999999999 2024-09-13T11:32:36.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111729+0000", 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140022+0000", 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028292571999999998 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140071+0000", 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9669000000000002e-05 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2800577870000005, 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028930938999999999, 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.423 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111269+0000", 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5700000000000002e-07 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113854+0000", 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025854340000000002 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113866+0000", 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1884e-05 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113873+0000", 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116175+0000", 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023019 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116218+0000", 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2796000000000002e-05 2024-09-13T11:32:36.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140165+0000", 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023947718999999999 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140199+0000", 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3596000000000002e-05 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:218 3.1 e38/30 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'7, mlcod=38'2)", 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2701948789999999, 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025863384999999999, 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121132+0000", 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3260000000000001e-06 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124734+0000", 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036019379999999998 2024-09-13T11:32:36.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124748+0000", 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913999999999999e-05 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124760+0000", 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1786e-05 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124849+0000", 2024-09-13T11:32:36.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8882999999999996e-05 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124874+0000", 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5466999999999999e-05 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146945+0000", 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022070837999999999 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146994+0000", 2024-09-13T11:32:36.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9233000000000001e-05 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2632616960000007, 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020872166000000001, 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128066+0000", 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.86e-06 2024-09-13T11:32:36.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131456+0000", 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033901360000000002 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131467+0000", 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0806e-05 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131474+0000", 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131524+0000", 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9262999999999999e-05 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131554+0000", 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0148999999999999e-05 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148883+0000", 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017329445999999998 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148936+0000", 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2879e-05 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.436 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2598234969999993, 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017580887, 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131503+0000", 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133995+0000", 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024920630000000001 2024-09-13T11:32:36.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134006+0000", 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0696000000000001e-05 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134012+0000", 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136541+0000", 2024-09-13T11:32:36.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025286150000000001 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136563+0000", 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2531000000000002e-05 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149031+0000", 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012467951 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149083+0000", 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2040999999999999e-05 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2336149190000008, 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022274233000000001, 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.441 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157712+0000", 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3689999999999999e-06 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161974+0000", 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042619160000000001 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161990+0000", 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5506e-05 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161998+0000", 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7520000000000002e-06 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162103+0000", 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104415 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162126+0000", 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3643e-05 2024-09-13T11:32:36.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179932+0000", 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017805476000000001 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179985+0000", 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3155999999999997e-05 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.445 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2292896859999995, 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018684003000000001, 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162037+0000", 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.083e-06 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166161+0000", 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041238799999999999 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166179+0000", 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8329999999999999e-05 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166190+0000", 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0822000000000001e-05 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166302+0000", 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011233700000000001 2024-09-13T11:32:36.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166328+0000", 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5185999999999999e-05 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180646+0000", 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014318261000000001 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180720+0000", 2024-09-13T11:32:36.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2251106840000006, 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014657148, 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:32:36.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1300000000000001e-07 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169754+0000", 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035389779999999999 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169772+0000", 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8020999999999999e-05 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0037999999999999e-05 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169847+0000", 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4695000000000003e-05 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169865+0000", 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7365999999999999e-05 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180834+0000", 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010969016 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180872+0000", 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8621000000000002e-05 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:32:36.454 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2131995310000008, 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018123856000000001, 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178128+0000", 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181360+0000", 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032324699999999999 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181371+0000", 2024-09-13T11:32:36.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0978000000000001e-05 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181378+0000", 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.861e-06 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181431+0000", 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3173000000000001e-05 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181455+0000", 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4236999999999999e-05 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196213+0000", 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01475807 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196250+0000", 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6636e-05 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.464 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2099159739999994, 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016397140000000001, 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181411+0000", 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7399999999999991e-07 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184468+0000", 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030575759999999998 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184481+0000", 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3072e-05 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184491+0000", 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184550+0000", 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8251999999999997e-05 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184575+0000", 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689000000000001e-05 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197720+0000", 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013144708999999999 2024-09-13T11:32:36.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197807+0000", 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6799999999999996e-05 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:245 3.1 e38/30 3:adf25eb4:::smithi20438004-46 low_tier:head v 38'15, mlcod=38'11)", 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.195673+0000", 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.1956523850000007, 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0079787639999999993, 2024-09-13T11:32:36.469 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.195673+0000", 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.195673+0000", 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.195675+0000", 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.341e-06 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.199719+0000", 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040446759999999997 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.199734+0000", 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4781e-05 2024-09-13T11:32:36.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.199749+0000", 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4999e-05 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.199829+0000", 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0196000000000003e-05 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.199876+0000", 2024-09-13T11:32:36.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6864999999999999e-05 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203596+0000", 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0037196239999999999 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203652+0000", 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6282000000000001e-05 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.473 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:249 3.1 e38/30 3:81faf980:::smithi20438004-50 low_tier:head v 38'21, mlcod=38'18)", 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.223177+0000", 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.1681489500000009, 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0088125389999999994, 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223177+0000", 2024-09-13T11:32:36.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223177+0000", 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223177+0000", 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0399999999999995e-07 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226670+0000", 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034928559999999999 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226683+0000", 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2884999999999999e-05 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226692+0000", 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2229999999999996e-06 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226793+0000", 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000100189 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226812+0000", 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9190000000000001e-05 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.231946+0000", 2024-09-13T11:32:36.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051346769999999998 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.231989+0000", 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2815000000000003e-05 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:281 4.a 4:56ef4fd4:::smithi20438004-32:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.1588772620000007, 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018358335999999999, 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232449+0000", 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7499999999999998e-07 2024-09-13T11:32:36.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232457+0000", 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3610000000000003e-06 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232470+0000", 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3230999999999999e-05 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232478+0000", 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5989999999999996e-06 2024-09-13T11:32:36.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232541+0000", 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3717e-05 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232619+0000", 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248106+0000", 2024-09-13T11:32:36.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015486616 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248991+0000", 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088509199999999998 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249363+0000", 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037176599999999999 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250719+0000", 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001356255 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250765+0000", 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6053000000000001e-05 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250807+0000", 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1675000000000003e-05 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.11.21:2 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'30, mlcod=38'26)", 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.240006+0000", 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.1513201839999994, 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076417769999999998, 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240006+0000", 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240006+0000", 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240006+0000", 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4600000000000004e-07 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240014+0000", 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8839999999999994e-06 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240021+0000", 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1749999999999999e-06 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240027+0000", 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7899999999999996e-06 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240084+0000", 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6626999999999998e-05 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240098+0000", 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4037e-05 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.247553+0000", 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007455766 2024-09-13T11:32:36.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.247647+0000", 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3851999999999995e-05 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:575 4.a 4:532cb07c:::smithi20438004-26:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.306336+0000", 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.0849901800000001, 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092199159999999999, 2024-09-13T11:32:36.489 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 575 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306336+0000", 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306336+0000", 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306337+0000", 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0240000000000001e-06 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306343+0000", 2024-09-13T11:32:36.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999999e-06 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306354+0000", 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.183e-05 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306362+0000", 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2910000000000002e-06 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306405+0000", 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3071000000000001e-05 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.306455+0000", 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0034999999999998e-05 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.311094+0000", 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046396099999999997 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.311803+0000", 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070887299999999995 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.311892+0000", 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8880000000000003e-05 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.315473+0000", 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003580443 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.315516+0000", 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3763000000000001e-05 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.315555+0000", 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8986000000000002e-05 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.495 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:36.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:32:36.498 INFO:tasks.rados.rados.0.smithi204.stdout:800: finishing set_chunk to oid 6 2024-09-13T11:32:36.498 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 184832 length: 1144832 2024-09-13T11:32:36.498 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:36.498 INFO:tasks.rados.rados.0.smithi204.stdout:803: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-13T11:32:36.498 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1183298 2024-09-13T11:32:36.498 INFO:tasks.rados.rados.0.smithi204.stdout:807: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:36.498 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:36.495+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:36.510 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:36.513 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:32:36.546 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2110812 2024-09-13T11:32:36.547 INFO:tasks.rados.rados.0.smithi204.stdout:801: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-13T11:32:36.570 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 818702 2024-09-13T11:32:36.598 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:36.596+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:36.615 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:36.616 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:36.616 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:36.616 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:36.620 INFO:tasks.rados.rados.0.smithi204.stdout:808: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-13T11:32:36.621 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:36.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:32:36.687 INFO:tasks.rados.rados.0.smithi204.stdout:802: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:36.688 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:36.699 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:36.698+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:36.757 INFO:tasks.rados.rados.0.smithi204.stdout:805: got (22) Invalid argument 2024-09-13T11:32:36.757 INFO:tasks.rados.rados.0.smithi204.stdout:809: finishing write tid 1 to smithi20438004-37 2024-09-13T11:32:36.757 INFO:tasks.rados.rados.0.smithi204.stdout:809: finishing write tid 2 to smithi20438004-37 2024-09-13T11:32:36.757 INFO:tasks.rados.rados.0.smithi204.stdout:809: finishing write tid 3 to smithi20438004-37 2024-09-13T11:32:36.757 INFO:tasks.rados.rados.0.smithi204.stdout:804: got (0) Success 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:809: finishing write tid 4 to smithi20438004-37 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:796: done (15 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:797: done (14 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:798: done (13 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:800: done (12 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:801: done (11 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:802: done (10 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:803: done (9 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:804: done (8 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:805: done (7 left) 2024-09-13T11:32:36.758 INFO:tasks.rados.rados.0.smithi204.stdout:806: done (6 left) 2024-09-13T11:32:36.759 INFO:tasks.rados.rados.0.smithi204.stdout:807: done (5 left) 2024-09-13T11:32:36.759 INFO:tasks.rados.rados.0.smithi204.stdout:808: done (4 left) 2024-09-13T11:32:36.759 INFO:tasks.rados.rados.0.smithi204.stdout:813: write (excl) oid 5 current snap is 0 2024-09-13T11:32:36.759 INFO:tasks.rados.rados.0.smithi204.stdout:813: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-13T11:32:36.763 INFO:tasks.rados.rados.0.smithi204.stdout:813: writing smithi20438004-5 from 749612 to 1223791 tid 1 2024-09-13T11:32:36.769 INFO:tasks.rados.rados.0.smithi204.stdout:813: writing smithi20438004-5 from 1954970 to 2386007 tid 2 2024-09-13T11:32:36.770 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:36.770 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:36.770 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:36.770 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:36.770 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:36.775 INFO:tasks.rados.rados.0.smithi204.stdout:813: writing smithi20438004-5 from 2919420 to 3535797 tid 3 2024-09-13T11:32:36.776 INFO:tasks.rados.rados.0.smithi204.stdout:814: write (excl) oid 13 current snap is 0 2024-09-13T11:32:36.776 INFO:tasks.rados.rados.0.smithi204.stdout:814: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-13T11:32:36.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:32:36.783 INFO:tasks.rados.rados.0.smithi204.stdout:814: writing smithi20438004-13 from 766419 to 1315847 tid 1 2024-09-13T11:32:36.790 INFO:tasks.rados.rados.0.smithi204.stdout:814: writing smithi20438004-13 from 2013452 to 2588147 tid 2 2024-09-13T11:32:36.797 INFO:tasks.rados.rados.0.smithi204.stdout:814: writing smithi20438004-13 from 3346843 to 3774492 tid 3 2024-09-13T11:32:36.797 INFO:tasks.rados.rados.0.smithi204.stdout:815: read oid 15 snap -1 2024-09-13T11:32:36.797 INFO:tasks.rados.rados.0.smithi204.stdout:815: expect deleted 2024-09-13T11:32:36.797 INFO:tasks.rados.rados.0.smithi204.stdout:816: set_chunk oid 42 target oid 2024-09-13T11:32:36.800 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:36.799+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:36.804 INFO:tasks.rados.rados.0.smithi204.stdout:816: set_chunk oid 42 offset: 2046976 length: 357376 target oid ad82002ed149d9318bcd076e62437014f68858368244f9bfdfcdf58ba2d04ba7 offset: 0 snap -1 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:817: tier_flush oid 24 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:817: tier_flush oid 24 snap -1 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:809: finishing write tid 5 to smithi20438004-37 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:809: finishing write tid 6 to smithi20438004-37 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:809: oid 37 updating version 0 to 28 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:809: oid 37 version 28 is already newer than 26 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:809: oid 37 version 28 is already newer than 27 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:809: oid 37 version 28 is already newer than 24 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:809: oid 37 version 28 is already newer than 25 2024-09-13T11:32:36.805 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 28 (ObjNum 114 snap 0 seq_num 114) dirty exists 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:809: left oid 37 (ObjNum 114 snap 0 seq_num 114) 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:810: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:810: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:809: done (8 left) 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:818: set_chunk oid 36 target oid 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:818: set_chunk oid 36 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:819: write oid 27 current snap is 0 2024-09-13T11:32:36.806 INFO:tasks.rados.rados.0.smithi204.stdout:819: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-13T11:32:36.811 INFO:tasks.rados.rados.0.smithi204.stdout:819: writing smithi20438004-27 from 783226 to 1407903 tid 1 2024-09-13T11:32:36.818 INFO:tasks.rados.rados.0.smithi204.stdout:819: writing smithi20438004-27 from 2071934 to 2506640 tid 2 2024-09-13T11:32:36.824 INFO:tasks.rados.rados.0.smithi204.stdout:819: writing smithi20438004-27 from 2974266 to 3613187 tid 3 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:820: write (excl) oid 47 current snap is 0 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: finishing write tid 5 to smithi20438004-17 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: oid 17 updating version 0 to 45 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: oid 17 updating version 45 to 46 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: oid 17 updating version 46 to 47 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: oid 17 version 47 is already newer than 44 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 47 (ObjNum 115 snap 0 seq_num 115) dirty exists 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:810: left oid 17 (ObjNum 115 snap 0 seq_num 115) 2024-09-13T11:32:36.825 INFO:tasks.rados.rados.0.smithi204.stdout:820: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-13T11:32:36.830 INFO:tasks.rados.rados.0.smithi204.stdout:820: writing smithi20438004-47 from 400033 to 1099959 tid 1 2024-09-13T11:32:36.836 INFO:tasks.rados.rados.0.smithi204.stdout:820: writing smithi20438004-47 from 1614063 to 2000033 tid 2 2024-09-13T11:32:36.837 INFO:tasks.rados.rados.0.smithi204.stdout:810: done (10 left) 2024-09-13T11:32:36.837 INFO:tasks.rados.rados.0.smithi204.stdout:811: done (9 left) 2024-09-13T11:32:36.837 INFO:tasks.rados.rados.0.smithi204.stdout:821: read oid 4 snap -1 2024-09-13T11:32:36.837 INFO:tasks.rados.rados.0.smithi204.stdout:821: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:36.837 INFO:tasks.rados.rados.0.smithi204.stdout:822: read oid 17 snap -1 2024-09-13T11:32:36.837 INFO:tasks.rados.rados.0.smithi204.stdout:822: expect (ObjNum 115 snap 0 seq_num 115) 2024-09-13T11:32:36.837 INFO:tasks.rados.rados.0.smithi204.stdout:823: tier_promote oid 43 2024-09-13T11:32:36.838 INFO:tasks.rados.rados.0.smithi204.stdout:812: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:36.838 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1 4.e 4:76a15ccf:::smithi20438004-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:26.453813+0000", 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 10.442984415, 2024-09-13T11:32:36.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078648060000000002, 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.453813+0000", 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.453813+0000", 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.453815+0000", 2024-09-13T11:32:36.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.719e-06 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.454241+0000", 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042630100000000002 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.454292+0000", 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1100999999999997e-05 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.454341+0000", 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8473000000000002e-05 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.455776+0000", 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001435101 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.455923+0000", 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014707500000000001 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.456111+0000", 2024-09-13T11:32:36.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018835600000000001 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.457670+0000", 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015591719999999999 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.458361+0000", 2024-09-13T11:32:36.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00069106899999999999 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.461601+0000", 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032398520000000001 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.461645+0000", 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3387000000000003e-05 2024-09-13T11:32:36.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:26.461678+0000", 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3200000000000001e-05 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:36.907 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.8054190249999991, 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:32:36.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:32:36.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7951444579999993, 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.913 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:32:36.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:32:36.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:32:36.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7824765570000007, 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:32:36.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7758667100000007, 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:32:36.923 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:32:36.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:32:36.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:32:36.928 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.928 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7674896800000006, 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:32:36.930 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:32:36.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:32:36.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7482276329999991, 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010735344000000001, 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148571+0000", 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:32:36.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154390+0000", 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058191809999999997 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154404+0000", 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4584e-05 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154415+0000", 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0661e-05 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154478+0000", 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2528999999999997e-05 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154502+0000", 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4460999999999998e-05 2024-09-13T11:32:36.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159243+0000", 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047404259999999998 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159305+0000", 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2582999999999994e-05 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.742358501, 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:36.944 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7334572230000003, 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:32:36.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7258595000000003, 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.949 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:32:36.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7204555710000005, 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:32:36.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.13.23:20 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.186798+0000", 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7099990250000001, 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010412591000000001, 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.958 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.186798+0000", 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.186798+0000", 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.186799+0000", 2024-09-13T11:32:36.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9599999999999998e-07 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.186805+0000", 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4049999999999999e-06 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.186815+0000", 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9480000000000003e-06 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.186827+0000", 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2809999999999999e-05 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.191311+0000", 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044837490000000004 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.191481+0000", 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017010800000000001 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197211+0000", 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005729675 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.7078318810000006, 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.962 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:32:36.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:32:36.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:32:36.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:36.967 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.6936524310000003, 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:32:36.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:32:36.971 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.6854342629999994, 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.972 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:32:36.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:32:36.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:32:36.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:295 4.e 4:7984cf84:::smithi20438004-46:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.240365+0000", 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.6564319520000002, 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010006787, 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.978 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 295 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240365+0000", 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240365+0000", 2024-09-13T11:32:36.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240367+0000", 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0839999999999999e-06 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240372+0000", 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0590000000000002e-06 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240385+0000", 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3145e-05 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240390+0000", 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2700000000000004e-06 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240438+0000", 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8384000000000003e-05 2024-09-13T11:32:36.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.240527+0000", 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8472000000000003e-05 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248238+0000", 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0077112489999999999 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249337+0000", 2024-09-13T11:32:36.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010988809999999999 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249554+0000", 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021676900000000001 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250283+0000", 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000729644 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250329+0000", 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6025000000000001e-05 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250372+0000", 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2805000000000001e-05 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.984 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.65461086, 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:32:36.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:32:36.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:32:36.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:298 4.c 4:3a6b347b:::smithi20438004-49:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242839+0000", 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.6539582779999993, 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010413649000000001, 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.991 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 298 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242839+0000", 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242839+0000", 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5799999999999998e-07 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242844+0000", 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3989999999999997e-06 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:36.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242862+0000", 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8201000000000001e-05 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242869+0000", 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7120000000000003e-06 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242917+0000", 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7948999999999998e-05 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242994+0000", 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6650999999999995e-05 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,12", 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249456+0000", 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064623129999999999 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250061+0000", 2024-09-13T11:32:36.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060531400000000003 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250256+0000", 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019423699999999999 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253177+0000", 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029211020000000001 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253216+0000", 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9613000000000002e-05 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253253+0000", 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6399999999999997e-05 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:36.997 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5078960729999995, 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:36.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:36.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:32:37.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.80272402099999995, 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:37.002 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:37.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:32:37.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:32:37.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:37.006 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:37.008 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:37.006+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:37.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:32:37.032 INFO:tasks.rados.rados.0.smithi204.stdout:813: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:37.032 INFO:tasks.rados.rados.0.smithi204.stdout:813: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:37.032 INFO:tasks.rados.rados.0.smithi204.stdout:813: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:37.032 INFO:tasks.rados.rados.0.smithi204.stdout:813: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:37.033 INFO:tasks.rados.rados.0.smithi204.stdout:813: finishing write tid 5 to smithi20438004-5 2024-09-13T11:32:37.033 INFO:tasks.rados.rados.0.smithi204.stdout:813: oid 5 updating version 0 to 30 2024-09-13T11:32:37.033 INFO:tasks.rados.rados.0.smithi204.stdout:813: oid 5 updating version 30 to 33 2024-09-13T11:32:37.033 INFO:tasks.rados.rados.0.smithi204.stdout:813: oid 5 version 33 is already newer than 32 2024-09-13T11:32:37.033 INFO:tasks.rados.rados.0.smithi204.stdout:813: oid 5 version 33 is already newer than 31 2024-09-13T11:32:37.033 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 33 (ObjNum 116 snap 0 seq_num 116) dirty exists 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:813: left oid 5 (ObjNum 116 snap 0 seq_num 116) 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:814: finishing write tid 1 to smithi20438004-13 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:812: done (11 left) 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:813: done (10 left) 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:824: read oid 39 snap -1 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:824: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:825: tier_promote oid 46 2024-09-13T11:32:37.034 INFO:tasks.rados.rados.0.smithi204.stdout:814: finishing write tid 2 to smithi20438004-13 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:814: finishing write tid 3 to smithi20438004-13 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:814: finishing write tid 4 to smithi20438004-13 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:818: finishing set_chunk to oid 36 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:818: got expected ENOENT (src dne) 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:816: finishing set_chunk to oid 42 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 42 set_chunk returned error code -95 offset: 2046976 length: 357376 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:814: finishing write tid 5 to smithi20438004-13 2024-09-13T11:32:37.035 INFO:tasks.rados.rados.0.smithi204.stdout:814: oid 13 updating version 0 to 31 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:814: oid 13 updating version 31 to 32 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:814: oid 13 updating version 32 to 34 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:814: oid 13 version 34 is already newer than 33 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 34 (ObjNum 117 snap 0 seq_num 117) dirty exists 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:814: left oid 13 (ObjNum 117 snap 0 seq_num 117) 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:819: finishing write tid 1 to smithi20438004-27 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:814: done (11 left) 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:815: done (10 left) 2024-09-13T11:32:37.036 INFO:tasks.rados.rados.0.smithi204.stdout:816: done (9 left) 2024-09-13T11:32:37.037 INFO:tasks.rados.rados.0.smithi204.stdout:818: done (8 left) 2024-09-13T11:32:37.037 INFO:tasks.rados.rados.0.smithi204.stdout:826: read oid 31 snap -1 2024-09-13T11:32:37.037 INFO:tasks.rados.rados.0.smithi204.stdout:826: expect (ObjNum 95 snap 0 seq_num 95) 2024-09-13T11:32:37.037 INFO:tasks.rados.rados.0.smithi204.stdout:827: read oid 38 snap -1 2024-09-13T11:32:37.037 INFO:tasks.rados.rados.0.smithi204.stdout:827: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-13T11:32:37.037 INFO:tasks.rados.rados.0.smithi204.stdout:828: set_chunk oid 30 target oid 2024-09-13T11:32:37.061 INFO:tasks.rados.rados.0.smithi204.stdout:828: set_chunk oid 30 offset: 863744 length: 2509824 target oid 52d9a55430821b68701da06e0c386fa420c2a386222572b3d00ce8b7e769faca offset: 0 snap -1 2024-09-13T11:32:37.062 INFO:tasks.rados.rados.0.smithi204.stdout:829: write (excl) oid 40 current snap is 0 2024-09-13T11:32:37.062 INFO:tasks.rados.rados.0.smithi204.stdout:829: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-13T11:32:37.067 INFO:tasks.rados.rados.0.smithi204.stdout:829: writing smithi20438004-40 from 416840 to 1192015 tid 1 2024-09-13T11:32:37.072 INFO:tasks.rados.rados.0.smithi204.stdout:829: writing smithi20438004-40 from 1672545 to 2016840 tid 2 2024-09-13T11:32:37.072 INFO:tasks.rados.rados.0.smithi204.stdout:830: read oid 23 snap -1 2024-09-13T11:32:37.072 INFO:tasks.rados.rados.0.smithi204.stdout:830: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:37.072 INFO:tasks.rados.rados.0.smithi204.stdout:831: write (excl) oid 16 current snap is 0 2024-09-13T11:32:37.073 INFO:tasks.rados.rados.0.smithi204.stdout:817: got (0) Success 2024-09-13T11:32:37.073 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-13T11:32:37.073 INFO:tasks.rados.rados.0.smithi204.stdout:819: finishing write tid 2 to smithi20438004-27 2024-09-13T11:32:37.073 INFO:tasks.rados.rados.0.smithi204.stdout:831: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-13T11:32:37.077 INFO:tasks.rados.rados.0.smithi204.stdout:831: writing smithi20438004-16 from 433647 to 884071 tid 1 2024-09-13T11:32:37.083 INFO:tasks.rados.rados.0.smithi204.stdout:831: writing smithi20438004-16 from 1331027 to 1913060 tid 2 2024-09-13T11:32:37.083 INFO:tasks.rados.rados.0.smithi204.stdout:817: done (13 left) 2024-09-13T11:32:37.083 INFO:tasks.rados.rados.0.smithi204.stdout:832: set_chunk oid 5 target oid 2024-09-13T11:32:37.108 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:37.107+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:37.111 INFO:tasks.rados.rados.0.smithi204.stdout:832: set_chunk oid 5 offset: 100352 length: 3489792 target oid 574e186947520a2389cbaeea8ba1176542b7ebf6bc2d1924ea8fea4e9a7674ef offset: 0 snap -1 2024-09-13T11:32:37.111 INFO:tasks.rados.rados.0.smithi204.stdout:833: tier_promote oid 9 2024-09-13T11:32:37.111 INFO:tasks.rados.rados.0.smithi204.stdout:819: finishing write tid 3 to smithi20438004-27 2024-09-13T11:32:37.111 INFO:tasks.rados.rados.0.smithi204.stdout:834: read oid 13 snap -1 2024-09-13T11:32:37.111 INFO:tasks.rados.rados.0.smithi204.stdout:834: expect (ObjNum 117 snap 0 seq_num 117) 2024-09-13T11:32:37.111 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: finishing write tid 4 to smithi20438004-27 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: finishing write tid 5 to smithi20438004-27 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: oid 27 updating version 0 to 60 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: oid 27 version 60 is already newer than 59 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: oid 27 updating version 60 to 61 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: oid 27 updating version 61 to 62 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 62 (ObjNum 118 snap 0 seq_num 118) dirty exists 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: left oid 27 (ObjNum 118 snap 0 seq_num 118) 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:820: finishing write tid 1 to smithi20438004-47 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:820: finishing write tid 2 to smithi20438004-47 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:819: done (15 left) 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:835: read oid 48 snap -1 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout:835: expect (ObjNum 79 snap 0 seq_num 79) 2024-09-13T11:32:37.112 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:820: finishing write tid 3 to smithi20438004-47 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:820: finishing write tid 4 to smithi20438004-47 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:820: oid 47 updating version 0 to 29 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:820: oid 47 updating version 29 to 30 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:820: oid 47 updating version 30 to 31 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 31 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:820: left oid 47 (ObjNum 119 snap 0 seq_num 119) 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:821: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:37.113 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:37.133 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:37.133 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:37.133 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:37.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:32:37.156 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:37.179 INFO:tasks.rados.rados.0.smithi204.stdout:822: expect (ObjNum 115 snap 0 seq_num 115) 2024-09-13T11:32:37.179 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:37.210 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:37.209+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:37.250 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:37.250 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:37.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:32:37.263 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1345317 2024-09-13T11:32:37.310 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:37.309+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:37.369 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'23)", 2024-09-13T11:32:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.586783185999998, 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005604375, 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779709+0000", 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.63e-07 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779721+0000", 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.204e-05 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779728+0000", 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4150000000000002e-06 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779732+0000", 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6839999999999998e-06 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779761+0000", 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9405999999999999e-05 2024-09-13T11:32:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779773+0000", 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1398e-05 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785264+0000", 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054913089999999998 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785313+0000", 2024-09-13T11:32:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8659999999999998e-05 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:44 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'26, mlcod=24'23)", 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.586213098999998, 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055622880000000003, 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780279+0000", 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-09-13T11:32:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780365+0000", 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6462e-05 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780372+0000", 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780376+0000", 2024-09-13T11:32:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784152+0000", 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037767650000000001 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784173+0000", 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0899000000000001e-05 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785814+0000", 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001640561 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785841+0000", 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6628e-05 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:45 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'27, mlcod=24'23)", 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.585625661999998, 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005510928, 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4999999999999998e-07 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780870+0000", 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6940000000000001e-06 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780877+0000", 2024-09-13T11:32:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0249999999999997e-06 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780883+0000", 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784553+0000", 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036697689999999998 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784573+0000", 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9936999999999999e-05 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786355+0000", 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017819870000000001 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786377+0000", 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1863000000000001e-05 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:46 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'28, mlcod=24'23)", 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.585030400999997, 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049328109999999996, 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781461+0000", 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7800000000000002e-07 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781471+0000", 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.533e-06 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781477+0000", 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2090000000000002e-06 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.781483+0000", 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4770000000000002e-06 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784808+0000", 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033250839999999999 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784826+0000", 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8161000000000001e-05 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786379+0000", 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001552905 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786394+0000", 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5064e-05 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:91 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'65, mlcod=24'61)", 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.384181398000003, 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053956129999999996, 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3700000000000001e-07 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982314+0000", 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2229999999999999e-06 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982320+0000", 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6499999999999999e-06 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982324+0000", 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6780000000000002e-06 2024-09-13T11:32:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.984345+0000", 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020208819999999999 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.984364+0000", 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8834e-05 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.987671+0000", 2024-09-13T11:32:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033072689999999998 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.987706+0000", 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.464e-05 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.629311688000001, 2024-09-13T11:32:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:32:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:32:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'105, mlcod=24'100)", 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.628748085000002, 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061709499999999997, 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737744+0000", 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7700000000000004e-07 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737749+0000", 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737756+0000", 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7649999999999996e-06 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737760+0000", 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6650000000000001e-06 2024-09-13T11:32:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742613+0000", 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048528909999999998 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742629+0000", 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6481e-05 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743902+0000", 2024-09-13T11:32:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012723630000000001 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743914+0000", 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2663999999999999e-05 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:140 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'106, mlcod=25'101)", 2024-09-13T11:32:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.628111701999998, 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005547064, 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738382+0000", 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5909999999999998e-06 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738386+0000", 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5240000000000001e-06 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738392+0000", 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8660000000000002e-06 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738396+0000", 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6739999999999996e-06 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742884+0000", 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004487441 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742896+0000", 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1866e-05 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743915+0000", 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010193229999999999 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743927+0000", 2024-09-13T11:32:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1779e-05 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:141 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'107, mlcod=25'101)", 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.739053+0000", 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.6274379, 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050398209999999999, 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739053+0000", 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739053+0000", 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739055+0000", 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6270000000000001e-06 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739104+0000", 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8699000000000002e-05 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739128+0000", 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3975999999999999e-05 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739140+0000", 2024-09-13T11:32:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2027e-05 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743177+0000", 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004037311 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743187+0000", 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8660000000000001e-06 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744070+0000", 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00088299899999999998 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744093+0000", 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3315999999999998e-05 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:142 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'108, mlcod=25'101)", 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.739436+0000", 2024-09-13T11:32:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.627054939000001, 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049474050000000002, 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739436+0000", 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739436+0000", 2024-09-13T11:32:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739439+0000", 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0110000000000001e-06 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739443+0000", 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6389999999999998e-06 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739449+0000", 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4010000000000002e-06 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.739453+0000", 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3979999999999999e-06 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743719+0000", 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042658669999999996 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743738+0000", 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9113000000000001e-05 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744357+0000", 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061914000000000001 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.744384+0000", 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6835999999999999e-05 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.889386061, 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:32:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:32:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.884327503, 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:32:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:32:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.883903681, 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:32:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:32:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.883885637000001, 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.13.23:1 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.085933+0000", 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.28055788, 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050668559999999998, 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085933+0000", 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085933+0000", 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085934+0000", 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7199999999999997e-07 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085939+0000", 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7899999999999999e-06 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085948+0000", 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9020000000000005e-06 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085956+0000", 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.807e-06 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.088108+0000", 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021517509999999999 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.088241+0000", 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013326000000000001 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.091000+0000", 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002759474 2024-09-13T11:32:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.167644865, 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:32:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:32:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.153804076, 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:32:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:32:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:402 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.274893+0000", 2024-09-13T11:32:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.091598806, 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053214830000000001, 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 402 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274893+0000", 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274893+0000", 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274894+0000", 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274899+0000", 2024-09-13T11:32:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1599999999999997e-06 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274913+0000", 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4051e-05 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274921+0000", 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2709999999999992e-06 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.274980+0000", 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9045999999999997e-05 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.275062+0000", 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1589999999999996e-05 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.277565+0000", 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002503857 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.278344+0000", 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077808900000000004 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.278489+0000", 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014568599999999999 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.280097+0000", 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001607598 2024-09-13T11:32:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.280155+0000", 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8400000000000003e-05 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.280214+0000", 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8739000000000001e-05 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.054836101999999, 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:32:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:32:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:32:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.3.0:27 4.d e43/31 4:bca378c8:::smithi20438004-20:head v 43'26, mlcod=40'24)", 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:35.774485+0000", 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.592006453, 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052517329999999997, 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.774485+0000", 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.774485+0000", 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.774486+0000", 2024-09-13T11:32:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1689999999999999e-06 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.776969+0000", 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024826610000000002 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.776986+0000", 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7651000000000001e-05 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.776996+0000", 2024-09-13T11:32:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0022e-05 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.777058+0000", 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1863e-05 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.777075+0000", 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7198999999999999e-05 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.779674+0000", 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025987219999999999 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:35.779737+0000", 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2446000000000005e-05 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:37.473 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:37.475 INFO:tasks.rados.rados.0.smithi204.stdout:823: got (0) Success 2024-09-13T11:32:37.475 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-13T11:32:37.475 INFO:tasks.rados.rados.0.smithi204.stdout:820: done (15 left) 2024-09-13T11:32:37.476 INFO:tasks.rados.rados.0.smithi204.stdout:821: done (14 left) 2024-09-13T11:32:37.476 INFO:tasks.rados.rados.0.smithi204.stdout:822: done (13 left) 2024-09-13T11:32:37.476 INFO:tasks.rados.rados.0.smithi204.stdout:823: done (12 left) 2024-09-13T11:32:37.476 INFO:tasks.rados.rados.0.smithi204.stdout:836: tier_flush oid 28 2024-09-13T11:32:37.476 INFO:tasks.rados.rados.0.smithi204.stdout:836: tier_flush oid 28 snap -1 2024-09-13T11:32:37.476 INFO:tasks.rados.rados.0.smithi204.stdout:826: expect (ObjNum 95 snap 0 seq_num 95) 2024-09-13T11:32:37.476 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:37.473+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:37.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:32:37.575 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:37.574+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:37.585 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:37.585 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:37.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:32:37.601 INFO:tasks.rados.rados.0.smithi204.stdout:825: got (0) Success 2024-09-13T11:32:37.602 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:37.602 INFO:tasks.rados.rados.0.smithi204.stdout:824: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:37.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:37.641 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2946427 2024-09-13T11:32:37.680 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:37.679+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:37.701 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:37.702 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:37.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:32:37.762 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:37.763 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:37.771 INFO:tasks.rados.rados.0.smithi204.stdout:827: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-13T11:32:37.778 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1692496 2024-09-13T11:32:37.778 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:37.776+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:37.781 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:37.781 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:37.781 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:37.798 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:37.799 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:37.815 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:37.815 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:37.833 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:37.833 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:37.850 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:37.850 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:37.859 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:37.859 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:37.867 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:37.867 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:37.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:32:37.879 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:37.877+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:37.883 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:37.883 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:37.886 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:37.900 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:37.901 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:37.918 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:37.935 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:37.935 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:828: finishing set_chunk to oid 30 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 36 (ObjNum 93 snap 0 seq_num 93) dirty exists 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: finishing write tid 1 to smithi20438004-40 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: finishing write tid 2 to smithi20438004-40 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: finishing write tid 3 to smithi20438004-40 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: finishing write tid 4 to smithi20438004-40 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: oid 40 updating version 0 to 49 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: oid 40 version 49 is already newer than 48 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: oid 40 updating version 49 to 50 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 50 (ObjNum 120 snap 0 seq_num 120) dirty exists 2024-09-13T11:32:37.949 INFO:tasks.rados.rados.0.smithi204.stdout:829: left oid 40 (ObjNum 120 snap 0 seq_num 120) 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: finishing write tid 1 to smithi20438004-16 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: finishing write tid 2 to smithi20438004-16 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: finishing write tid 3 to smithi20438004-16 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: finishing write tid 4 to smithi20438004-16 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: oid 16 updating version 0 to 25 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: oid 16 updating version 25 to 27 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: oid 16 version 27 is already newer than 26 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 27 (ObjNum 121 snap 0 seq_num 121) dirty exists 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:831: left oid 16 (ObjNum 121 snap 0 seq_num 121) 2024-09-13T11:32:37.950 INFO:tasks.rados.rados.0.smithi204.stdout:830: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:37.953 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:37.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:37.971 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:37.971 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:37.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:37.971+0000 7f0cdffff640 0 -- 172.21.15.38:0/2805717470 >> v2:172.21.15.154:6804/538351706 conn(0x7f0ce021cfc0 msgr2=0x7f0ce021f3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:37.979 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:37.978+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:37.989 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:37.989 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:37.992 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:37.993 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:32:38.006 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:38.007 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:38.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:38.080 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:38.079+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:38.113 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:38.114 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:32:38.132 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:38.181 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:38.180+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:833: got (0) Success 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 32 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:824: done (12 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:825: done (11 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:826: done (10 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:827: done (9 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:828: done (8 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:829: done (7 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:830: done (6 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:831: done (5 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:833: done (4 left) 2024-09-13T11:32:38.233 INFO:tasks.rados.rados.0.smithi204.stdout:837: set_chunk oid 37 target oid 2024-09-13T11:32:38.253 INFO:tasks.rados.rados.0.smithi204.stdout:837: set_chunk oid 37 offset: 452608 length: 1594368 target oid 8c582d468ed9edb064b1ca18cabf9f4719d82fde91d78764186ee1aff3d9c206 offset: 0 snap -1 2024-09-13T11:32:38.253 INFO:tasks.rados.rados.0.smithi204.stdout:838: read oid 16 snap -1 2024-09-13T11:32:38.253 INFO:tasks.rados.rados.0.smithi204.stdout:838: expect (ObjNum 121 snap 0 seq_num 121) 2024-09-13T11:32:38.253 INFO:tasks.rados.rados.0.smithi204.stdout:839: read oid 19 snap -1 2024-09-13T11:32:38.254 INFO:tasks.rados.rados.0.smithi204.stdout:839: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:38.254 INFO:tasks.rados.rados.0.smithi204.stdout:840: set_chunk oid 49 target oid 2024-09-13T11:32:38.257 INFO:tasks.rados.rados.0.smithi204.stdout:840: set_chunk oid 49 offset: 2700800 length: 101376 target oid 250b7a8c7c2aff03751861c555b536d8d63c2dd0043b099655ad91bd2bada237 offset: 0 snap -1 2024-09-13T11:32:38.258 INFO:tasks.rados.rados.0.smithi204.stdout:841: read oid 33 snap -1 2024-09-13T11:32:38.258 INFO:tasks.rados.rados.0.smithi204.stdout:841: expect deleted 2024-09-13T11:32:38.258 INFO:tasks.rados.rados.0.smithi204.stdout:842: delete oid 27 current snap is 0 2024-09-13T11:32:38.258 INFO:tasks.rados.rados.0.smithi204.stdout:832: finishing set_chunk to oid 5 2024-09-13T11:32:38.258 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 33 (ObjNum 116 snap 0 seq_num 116) dirty exists 2024-09-13T11:32:38.258 INFO:tasks.rados.rados.0.smithi204.stdout:835: expect (ObjNum 79 snap 0 seq_num 79) 2024-09-13T11:32:38.258 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:38.270 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:38.271 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:32:38.281 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:38.280+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:38.310 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 347426 2024-09-13T11:32:38.357 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 310311 2024-09-13T11:32:38.382 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:38.381+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:38.387 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:38.387 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.395 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:32:38.404 INFO:tasks.rados.rados.0.smithi204.stdout:834: expect (ObjNum 117 snap 0 seq_num 117) 2024-09-13T11:32:38.404 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:38.484 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:38.482+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:38.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:32:38.528 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:38.543 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:38.543 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:32:38.590 INFO:tasks.rados.rados.0.smithi204.stdout:836: got (2) No such file or directory 2024-09-13T11:32:38.590 INFO:tasks.rados.rados.0.smithi204.stdout:836: got expected ENOENT (src dne) 2024-09-13T11:32:38.591 INFO:tasks.rados.rados.0.smithi204.stdout:837: finishing set_chunk to oid 37 2024-09-13T11:32:38.591 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 28 (ObjNum 114 snap 0 seq_num 114) dirty exists 2024-09-13T11:32:38.591 INFO:tasks.rados.rados.0.smithi204.stdout:840: finishing set_chunk to oid 49 2024-09-13T11:32:38.591 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 2700800 length: 101376 2024-09-13T11:32:38.591 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-13T11:32:38.591 INFO:tasks.rados.rados.0.smithi204.stdout:838: expect (ObjNum 121 snap 0 seq_num 121) 2024-09-13T11:32:38.591 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 79124 2024-09-13T11:32:38.593 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:38.592+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:38.614 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1288187 2024-09-13T11:32:38.647 INFO:tasks.rados.rados.0.smithi204.stdout:839: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:38.665 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1891135 2024-09-13T11:32:38.685 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:38.684+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:38.702 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:38.702 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:32:38.786 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:38.785+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:38.789 INFO:tasks.rados.rados.0.smithi204.stdout:832: done (9 left) 2024-09-13T11:32:38.789 INFO:tasks.rados.rados.0.smithi204.stdout:834: done (8 left) 2024-09-13T11:32:38.789 INFO:tasks.rados.rados.0.smithi204.stdout:835: done (7 left) 2024-09-13T11:32:38.789 INFO:tasks.rados.rados.0.smithi204.stdout:836: done (6 left) 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:837: done (5 left) 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:838: done (4 left) 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:839: done (3 left) 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:840: done (2 left) 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:841: done (1 left) 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:842: done (0 left) 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:843: write oid 5 current snap is 0 2024-09-13T11:32:38.790 INFO:tasks.rados.rados.0.smithi204.stdout:843: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-13T11:32:38.795 INFO:tasks.rados.rados.0.smithi204.stdout:843: writing smithi20438004-5 from 467261 to 1184536 tid 1 2024-09-13T11:32:38.801 INFO:tasks.rados.rados.0.smithi204.stdout:843: writing smithi20438004-5 from 1847991 to 2067261 tid 2 2024-09-13T11:32:38.801 INFO:tasks.rados.rados.0.smithi204.stdout:844: read oid 8 snap -1 2024-09-13T11:32:38.801 INFO:tasks.rados.rados.0.smithi204.stdout:844: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:38.801 INFO:tasks.rados.rados.0.smithi204.stdout:845: write oid 34 current snap is 0 2024-09-13T11:32:38.801 INFO:tasks.rados.rados.0.smithi204.stdout:845: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-13T11:32:38.807 INFO:tasks.rados.rados.0.smithi204.stdout:845: writing smithi20438004-34 from 484068 to 1276592 tid 1 2024-09-13T11:32:38.813 INFO:tasks.rados.rados.0.smithi204.stdout:845: writing smithi20438004-34 from 1906473 to 2084068 tid 2 2024-09-13T11:32:38.813 INFO:tasks.rados.rados.0.smithi204.stdout:846: write (excl) oid 17 current snap is 0 2024-09-13T11:32:38.813 INFO:tasks.rados.rados.0.smithi204.stdout:843: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:38.813 INFO:tasks.rados.rados.0.smithi204.stdout:843: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:38.813 INFO:tasks.rados.rados.0.smithi204.stdout:846: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-13T11:32:38.818 INFO:tasks.rados.rados.0.smithi204.stdout:846: writing smithi20438004-17 from 500875 to 968648 tid 1 2024-09-13T11:32:38.824 INFO:tasks.rados.rados.0.smithi204.stdout:846: writing smithi20438004-17 from 1564955 to 2100875 tid 2 2024-09-13T11:32:38.824 INFO:tasks.rados.rados.0.smithi204.stdout:847: set_chunk oid 20 target oid 2024-09-13T11:32:38.836 INFO:tasks.rados.rados.0.smithi204.stdout:847: set_chunk oid 20 offset: 297984 length: 1170432 target oid 20c12f9c4e19a2fabb9fda789f414a03b3d97e8a49a6155a5cbbd3d18f45ae00 offset: 0 snap -1 2024-09-13T11:32:38.836 INFO:tasks.rados.rados.0.smithi204.stdout:843: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:848: read oid 13 snap -1 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:848: expect (ObjNum 117 snap 0 seq_num 117) 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:849: read oid 4 snap -1 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:849: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:850: read oid 50 snap -1 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:850: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:851: read oid 28 snap -1 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:851: expect deleted 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:852: read oid 31 snap -1 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:852: expect (ObjNum 95 snap 0 seq_num 95) 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:853: tier_promote oid 41 2024-09-13T11:32:38.837 INFO:tasks.rados.rados.0.smithi204.stdout:843: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:38.838 INFO:tasks.rados.rados.0.smithi204.stdout:843: oid 5 updating version 0 to 36 2024-09-13T11:32:38.838 INFO:tasks.rados.rados.0.smithi204.stdout:843: oid 5 version 36 is already newer than 35 2024-09-13T11:32:38.838 INFO:tasks.rados.rados.0.smithi204.stdout:843: oid 5 updating version 36 to 37 2024-09-13T11:32:38.838 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 37 (ObjNum 123 snap 0 seq_num 123) dirty exists 2024-09-13T11:32:38.838 INFO:tasks.rados.rados.0.smithi204.stdout:843: left oid 5 (ObjNum 123 snap 0 seq_num 123) 2024-09-13T11:32:38.838 INFO:tasks.rados.rados.0.smithi204.stdout:844: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:38.838 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:38.843 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:38.843 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.854 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:32:38.868 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1010917 2024-09-13T11:32:38.887 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:38.886+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:38.905 INFO:tasks.rados.rados.0.smithi204.stdout:845: finishing write tid 1 to smithi20438004-34 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: finishing write tid 2 to smithi20438004-34 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: finishing write tid 3 to smithi20438004-34 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: finishing write tid 4 to smithi20438004-34 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: oid 34 updating version 0 to 74 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: oid 34 updating version 74 to 75 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: oid 34 version 75 is already newer than 73 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 75 (ObjNum 124 snap 0 seq_num 124) dirty exists 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: left oid 34 (ObjNum 124 snap 0 seq_num 124) 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:846: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:843: done (10 left) 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:844: done (9 left) 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:845: done (8 left) 2024-09-13T11:32:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:854: read oid 35 snap -1 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:854: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:855: read oid 26 snap -1 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:855: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:856: read oid 16 snap -1 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:856: expect (ObjNum 121 snap 0 seq_num 121) 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:857: read oid 7 snap -1 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:857: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:858: read oid 24 snap -1 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:858: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-13T11:32:38.907 INFO:tasks.rados.rados.0.smithi204.stdout:859: tier_evict oid 21 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:859: tier_evict oid 21 snap -1 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: oid 17 updating version 0 to 51 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: oid 17 updating version 51 to 52 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: oid 17 updating version 52 to 53 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 53 (ObjNum 125 snap 0 seq_num 125) dirty exists 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: left oid 17 (ObjNum 125 snap 0 seq_num 125) 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:847: finishing set_chunk to oid 20 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 297984 length: 1170432 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:846: done (13 left) 2024-09-13T11:32:38.908 INFO:tasks.rados.rados.0.smithi204.stdout:847: done (12 left) 2024-09-13T11:32:38.909 INFO:tasks.rados.rados.0.smithi204.stdout:860: set_chunk oid 30 target oid 2024-09-13T11:32:38.916 INFO:tasks.rados.rados.0.smithi204.stdout:860: set_chunk oid 30 offset: 2513920 length: 486912 target oid 6bf16f569cbfefb23bf5bedd7c28047571d8f15c91ff1cec60319acb2346c56e offset: 0 snap -1 2024-09-13T11:32:38.916 INFO:tasks.rados.rados.0.smithi204.stdout:861: read oid 43 snap -1 2024-09-13T11:32:38.916 INFO:tasks.rados.rados.0.smithi204.stdout:861: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:38.916 INFO:tasks.rados.rados.0.smithi204.stdout:862: read oid 2 snap -1 2024-09-13T11:32:38.916 INFO:tasks.rados.rados.0.smithi204.stdout:862: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:38.917 INFO:tasks.rados.rados.0.smithi204.stdout:863: write oid 42 current snap is 0 2024-09-13T11:32:38.917 INFO:tasks.rados.rados.0.smithi204.stdout:850: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:38.940 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2711354 2024-09-13T11:32:38.964 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:38.965 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:38.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:32:38.989 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:38.987+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:39.005 INFO:tasks.rados.rados.0.smithi204.stdout:852: expect (ObjNum 95 snap 0 seq_num 95) 2024-09-13T11:32:39.076 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.077 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.085 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:32:39.090 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:39.089+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:39.188 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.188 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.191 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:39.190+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:39.196 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:32:39.203 INFO:tasks.rados.rados.0.smithi204.stdout:853: got (0) Success 2024-09-13T11:32:39.203 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 39 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-09-13T11:32:39.203 INFO:tasks.rados.rados.0.smithi204.stdout:849: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:39.228 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1633167 2024-09-13T11:32:39.260 INFO:tasks.rados.rados.0.smithi204.stdout:848: expect (ObjNum 117 snap 0 seq_num 117) 2024-09-13T11:32:39.260 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:39.292 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:39.291+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:39.322 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:39.338 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.338 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:32:39.394 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:39.392+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:39.427 INFO:tasks.rados.rados.0.smithi204.stdout:859: got (22) Invalid argument 2024-09-13T11:32:39.427 INFO:tasks.rados.rados.0.smithi204.stdout:855: expect (ObjNum 68 snap 0 seq_num 68) 2024-09-13T11:32:39.449 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.449 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:32:39.460 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:39.500 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:39.499+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:39.525 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:32:39.530 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2778809 2024-09-13T11:32:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:856: expect (ObjNum 121 snap 0 seq_num 121) 2024-09-13T11:32:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:32:39.572 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1574812 2024-09-13T11:32:39.586 INFO:tasks.rados.rados.0.smithi204.stdout:858: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-13T11:32:39.611 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:39.610+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:39.652 INFO:tasks.rados.rados.0.smithi204.stdout:860: finishing set_chunk to oid 30 2024-09-13T11:32:39.652 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 2513920 length: 486912 2024-09-13T11:32:39.652 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 93 snap 0 seq_num 93) dirty exists 2024-09-13T11:32:39.652 INFO:tasks.rados.rados.0.smithi204.stdout:857: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-13T11:32:39.652 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:39.699 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:39.698+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:39.713 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.713 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.721 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:32:39.768 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3768986 2024-09-13T11:32:39.800 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:39.798+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:39.823 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.823 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:32:39.845 INFO:tasks.rados.rados.0.smithi204.stdout:854: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:39.900 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:39.899+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:39.911 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2629481 2024-09-13T11:32:39.975 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:39.975 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:39.984 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:32:39.993 INFO:tasks.rados.rados.0.smithi204.stdout:862: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:40.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:40.027 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:40.026+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:40.052 INFO:tasks.rados.rados.0.smithi204.stdout:861: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:40.052 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:40.086 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:40.087 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:40.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:32:40.101 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:40.100+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:40.148 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:40.148 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:40.167 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:40.167 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:40.177 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2959981 2024-09-13T11:32:40.183 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:40.183 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:40.199 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:40.199 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:40.203 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:40.202+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:40.204 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1780376 2024-09-13T11:32:40.218 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:40.218 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:40.236 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:40.236 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:40.240 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:40.240 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:40.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:32:40.252 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:40.252 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:40.254 INFO:tasks.rados.rados.0.smithi204.stdout:863: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-13T11:32:40.259 INFO:tasks.rados.rados.0.smithi204.stdout:863: writing smithi20438004-42 from 517682 to 1060704 tid 1 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:863: writing smithi20438004-42 from 1623437 to 2117682 tid 2 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:848: done (15 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:849: done (14 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:850: done (13 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:851: done (12 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:852: done (11 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:853: done (10 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:854: done (9 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:855: done (8 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:856: done (7 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:857: done (6 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:858: done (5 left) 2024-09-13T11:32:40.267 INFO:tasks.rados.rados.0.smithi204.stdout:859: done (4 left) 2024-09-13T11:32:40.268 INFO:tasks.rados.rados.0.smithi204.stdout:860: done (3 left) 2024-09-13T11:32:40.268 INFO:tasks.rados.rados.0.smithi204.stdout:861: done (2 left) 2024-09-13T11:32:40.268 INFO:tasks.rados.rados.0.smithi204.stdout:862: done (1 left) 2024-09-13T11:32:40.268 INFO:tasks.rados.rados.0.smithi204.stdout:864: set_chunk oid 10 target oid 2024-09-13T11:32:40.270 INFO:tasks.rados.rados.0.smithi204.stdout:864: set_chunk oid 10 offset: 1450496 length: 210944 target oid 7855d650e60c5313b4a34bf8d62fbf7d2dfc61c3c67166961597447c82a0e937 offset: 0 snap -1 2024-09-13T11:32:40.270 INFO:tasks.rados.rados.0.smithi204.stdout:865: set_chunk oid 18 target oid 2024-09-13T11:32:40.271 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:40.271 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:865: set_chunk oid 18 offset: 2714112 length: 142336 target oid d2eba2f55a77008fa49f4e5c0ebcafe587dc1d9c9f805fb37678c7fbc337f6f4 offset: 0 snap -1 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:866: read oid 39 snap -1 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:866: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:867: read oid 3 snap -1 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:867: expect deleted 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:868: write (excl) oid 41 current snap is 0 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:863: finishing write tid 1 to smithi20438004-42 2024-09-13T11:32:40.274 INFO:tasks.rados.rados.0.smithi204.stdout:868: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-13T11:32:40.280 INFO:tasks.rados.rados.0.smithi204.stdout:868: writing smithi20438004-41 from 534489 to 1152760 tid 1 2024-09-13T11:32:40.286 INFO:tasks.rados.rados.0.smithi204.stdout:868: writing smithi20438004-41 from 1565566 to 2134489 tid 2 2024-09-13T11:32:40.286 INFO:tasks.rados.rados.0.smithi204.stdout:869: write oid 5 current snap is 0 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:863: finishing write tid 2 to smithi20438004-42 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:863: finishing write tid 3 to smithi20438004-42 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:863: finishing write tid 4 to smithi20438004-42 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:863: oid 42 updating version 0 to 30 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:863: oid 42 version 30 is already newer than 29 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:863: oid 42 updating version 30 to 31 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 31 (ObjNum 126 snap 0 seq_num 126) dirty exists 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:863: left oid 42 (ObjNum 126 snap 0 seq_num 126) 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:864: finishing set_chunk to oid 10 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 63 (ObjNum 100 snap 0 seq_num 100) dirty exists 2024-09-13T11:32:40.287 INFO:tasks.rados.rados.0.smithi204.stdout:869: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-13T11:32:40.289 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:40.289 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:40.293 INFO:tasks.rados.rados.0.smithi204.stdout:869: writing smithi20438004-5 from 551296 to 1244816 tid 1 2024-09-13T11:32:40.298 INFO:tasks.rados.rados.0.smithi204.stdout:869: writing smithi20438004-5 from 2024048 to 2151296 tid 2 2024-09-13T11:32:40.298 INFO:tasks.rados.rados.0.smithi204.stdout:863: done (6 left) 2024-09-13T11:32:40.298 INFO:tasks.rados.rados.0.smithi204.stdout:864: done (5 left) 2024-09-13T11:32:40.298 INFO:tasks.rados.rados.0.smithi204.stdout:870: write oid 25 current snap is 0 2024-09-13T11:32:40.298 INFO:tasks.rados.rados.0.smithi204.stdout:870: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-13T11:32:40.303 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:40.302+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:40.305 INFO:tasks.rados.rados.0.smithi204.stdout:870: writing smithi20438004-25 from 568103 to 1336872 tid 1 2024-09-13T11:32:40.306 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:40.306 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:40.310 INFO:tasks.rados.rados.0.smithi204.stdout:870: writing smithi20438004-25 from 2082530 to 2168103 tid 2 2024-09-13T11:32:40.310 INFO:tasks.rados.rados.0.smithi204.stdout:871: read oid 23 snap -1 2024-09-13T11:32:40.310 INFO:tasks.rados.rados.0.smithi204.stdout:871: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:40.310 INFO:tasks.rados.rados.0.smithi204.stdout:872: tier_promote oid 13 2024-09-13T11:32:40.311 INFO:tasks.rados.rados.0.smithi204.stdout:865: finishing set_chunk to oid 18 2024-09-13T11:32:40.311 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 43 (ObjNum 81 snap 0 seq_num 81) dirty exists 2024-09-13T11:32:40.311 INFO:tasks.rados.rados.0.smithi204.stdout:865: done (7 left) 2024-09-13T11:32:40.311 INFO:tasks.rados.rados.0.smithi204.stdout:867: done (6 left) 2024-09-13T11:32:40.311 INFO:tasks.rados.rados.0.smithi204.stdout:873: set_chunk oid 4 target oid 2024-09-13T11:32:40.320 INFO:tasks.rados.rados.0.smithi204.stdout:873: set_chunk oid 4 offset: 674816 length: 17408 target oid 89e8443bb4be7181cf82b056aa5ef7a682963cdae1f7eaec710ce9542e51c372 offset: 0 snap -1 2024-09-13T11:32:40.320 INFO:tasks.rados.rados.0.smithi204.stdout:874: set_chunk oid 9 target oid 2024-09-13T11:32:40.326 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:40.326 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:40.332 INFO:tasks.rados.rados.0.smithi204.stdout:874: set_chunk oid 9 offset: 497664 length: 1129472 target oid ddb57aca84dba6cc00fa1a3ff173f68b71a37ec1f1e664058e0a0a77951b0ff9 offset: 0 snap -1 2024-09-13T11:32:40.332 INFO:tasks.rados.rados.0.smithi204.stdout:875: write (excl) oid 38 current snap is 0 2024-09-13T11:32:40.333 INFO:tasks.rados.rados.0.smithi204.stdout:866: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-13T11:32:40.343 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:40.343 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:40.360 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:40.361 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:40.368 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:40.368 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:40.368 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:40.369 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:40.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:32:40.378 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:40.378 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:40.395 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:40.395 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:40.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:40.404 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:40.402+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:40.505 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:40.503+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:40.522 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:40.522 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:40.522 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:40.522 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:40.522 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:40.525 INFO:tasks.rados.rados.0.smithi204.stdout:868: finishing write tid 1 to smithi20438004-41 2024-09-13T11:32:40.525 INFO:tasks.rados.rados.0.smithi204.stdout:868: finishing write tid 2 to smithi20438004-41 2024-09-13T11:32:40.525 INFO:tasks.rados.rados.0.smithi204.stdout:868: finishing write tid 3 to smithi20438004-41 2024-09-13T11:32:40.525 INFO:tasks.rados.rados.0.smithi204.stdout:868: finishing write tid 4 to smithi20438004-41 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:868: oid 41 updating version 0 to 81 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:868: oid 41 version 81 is already newer than 80 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:868: oid 41 updating version 81 to 82 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 82 (ObjNum 127 snap 0 seq_num 127) dirty exists 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:868: left oid 41 (ObjNum 127 snap 0 seq_num 127) 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:869: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:869: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:869: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:40.526 INFO:tasks.rados.rados.0.smithi204.stdout:875: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-13T11:32:40.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:32:40.532 INFO:tasks.rados.rados.0.smithi204.stdout:875: writing smithi20438004-38 from 584910 to 1145281 tid 1 2024-09-13T11:32:40.538 INFO:tasks.rados.rados.0.smithi204.stdout:875: writing smithi20438004-38 from 1857365 to 2184910 tid 2 2024-09-13T11:32:40.538 INFO:tasks.rados.rados.0.smithi204.stdout:866: done (8 left) 2024-09-13T11:32:40.538 INFO:tasks.rados.rados.0.smithi204.stdout:868: done (7 left) 2024-09-13T11:32:40.538 INFO:tasks.rados.rados.0.smithi204.stdout:876: read oid 48 snap -1 2024-09-13T11:32:40.538 INFO:tasks.rados.rados.0.smithi204.stdout:876: expect (ObjNum 79 snap 0 seq_num 79) 2024-09-13T11:32:40.538 INFO:tasks.rados.rados.0.smithi204.stdout:877: write oid 7 current snap is 0 2024-09-13T11:32:40.538 INFO:tasks.rados.rados.0.smithi204.stdout:877: seq_num 131 ranges {601717=635620,1799494=402223} 2024-09-13T11:32:40.544 INFO:tasks.rados.rados.0.smithi204.stdout:877: writing smithi20438004-7 from 601717 to 1237337 tid 1 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:877: writing smithi20438004-7 from 1799494 to 2201717 tid 2 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:878: read oid 15 snap -1 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:878: expect deleted 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:879: write (excl) oid 28 current snap is 0 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:869: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:869: oid 5 updating version 0 to 39 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:869: oid 5 version 39 is already newer than 38 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:869: oid 5 updating version 39 to 40 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 40 (ObjNum 128 snap 0 seq_num 128) dirty exists 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:869: left oid 5 (ObjNum 128 snap 0 seq_num 128) 2024-09-13T11:32:40.550 INFO:tasks.rados.rados.0.smithi204.stdout:872: got (0) Success 2024-09-13T11:32:40.551 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 34 (ObjNum 117 snap 0 seq_num 117) dirty exists 2024-09-13T11:32:40.551 INFO:tasks.rados.rados.0.smithi204.stdout:870: finishing write tid 1 to smithi20438004-25 2024-09-13T11:32:40.551 INFO:tasks.rados.rados.0.smithi204.stdout:879: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-13T11:32:40.557 INFO:tasks.rados.rados.0.smithi204.stdout:879: writing smithi20438004-28 from 618524 to 1329393 tid 1 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:879: writing smithi20438004-28 from 1857976 to 2218524 tid 2 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:869: done (10 left) 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:872: done (9 left) 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:880: delete oid 49 current snap is 0 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:870: finishing write tid 2 to smithi20438004-25 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:870: finishing write tid 3 to smithi20438004-25 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:870: finishing write tid 4 to smithi20438004-25 2024-09-13T11:32:40.562 INFO:tasks.rados.rados.0.smithi204.stdout:870: oid 25 updating version 0 to 66 2024-09-13T11:32:40.563 INFO:tasks.rados.rados.0.smithi204.stdout:870: oid 25 version 66 is already newer than 64 2024-09-13T11:32:40.563 INFO:tasks.rados.rados.0.smithi204.stdout:870: oid 25 version 66 is already newer than 65 2024-09-13T11:32:40.563 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 66 (ObjNum 129 snap 0 seq_num 129) dirty exists 2024-09-13T11:32:40.563 INFO:tasks.rados.rados.0.smithi204.stdout:870: left oid 25 (ObjNum 129 snap 0 seq_num 129) 2024-09-13T11:32:40.563 INFO:tasks.rados.rados.0.smithi204.stdout:871: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:40.606 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:40.604+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:40.676 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:201 3.1 3:b265d015:::smithi20438004-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.589393962999999, 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030631999, 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.677 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 201 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084693+0000", 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4739999999999999e-06 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.884e-06 2024-09-13T11:32:40.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2858e-05 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084748+0000", 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3113e-05 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084807+0000", 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8523999999999997e-05 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084930+0000", 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012317099999999999 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097750+0000", 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012820575000000001 2024-09-13T11:32:40.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107462+0000", 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097120990000000001 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114842+0000", 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073796520000000004 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115192+0000", 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035007400000000002 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115249+0000", 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7277000000000001e-05 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115323+0000", 2024-09-13T11:32:40.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3298000000000002e-05 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:202 3.1 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.589312516, 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055756029999999998, 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.684 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084773+0000", 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084777+0000", 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3540000000000002e-06 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084784+0000", 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084789+0000", 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5480000000000001e-06 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085081+0000", 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000291673 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085197+0000", 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011550800000000001 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109819+0000", 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024622326999999999 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115554+0000", 2024-09-13T11:32:40.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057346100000000002 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.139296+0000", 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023742025999999999 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140477+0000", 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011813660000000001 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140507+0000", 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0307e-05 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140528+0000", 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0934e-05 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:203 3.1 3:970ba768:::smithi20438004-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:32:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.589269748, 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055588820999999997, 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1900000000000003e-07 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084824+0000", 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3110000000000006e-06 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084831+0000", 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4869999999999996e-06 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084839+0000", 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7840000000000004e-06 2024-09-13T11:32:40.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085318+0000", 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047915700000000002 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085404+0000", 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5576999999999997e-05 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.108042+0000", 2024-09-13T11:32:40.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022638703999999999 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115550+0000", 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075072150000000002 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132897+0000", 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017347621000000001 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140260+0000", 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073629139999999999 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140331+0000", 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0994999999999994e-05 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140404+0000", 2024-09-13T11:32:40.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2537e-05 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:206 3.1 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.589234126999999, 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022595217000000001, 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1799999999999995e-07 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084854+0000", 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1650000000000002e-06 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084861+0000", 2024-09-13T11:32:40.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1300000000000003e-06 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084868+0000", 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.55e-06 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085510+0000", 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064168099999999996 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085595+0000", 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5643999999999995e-05 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.091806+0000", 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062107220000000001 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107236+0000", 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015429897999999999 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107281+0000", 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4994000000000003e-05 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107302+0000", 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1163e-05 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107375+0000", 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2915000000000002e-05 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107446+0000", 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0837e-05 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:210 3.1 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.589206991999999, 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047980961000000003, 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.703 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 210 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084883+0000", 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084890+0000", 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0650000000000001e-06 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084896+0000", 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9000000000000003e-06 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085703+0000", 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000807097 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085822+0000", 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119098 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.104833+0000", 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019011389 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115543+0000", 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01070981 2024-09-13T11:32:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127458+0000", 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011915189 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132784+0000", 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053251139999999997 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132825+0000", 2024-09-13T11:32:40.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1329999999999999e-05 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132859+0000", 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:215 3.1 3:84f0de12:::smithi20438004-16 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.589180675, 2024-09-13T11:32:40.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034172806999999999, 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 215 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0400000000000002e-07 2024-09-13T11:32:40.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084907+0000", 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.666e-06 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084915+0000", 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9330000000000001e-06 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084920+0000", 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8289999999999998e-06 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085942+0000", 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001022029 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.086010+0000", 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8007000000000001e-05 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.100909+0000", 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014898991 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107467+0000", 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065583270000000001 2024-09-13T11:32:40.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.117910+0000", 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010443309 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.118971+0000", 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010605510000000001 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119023+0000", 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2114000000000003e-05 2024-09-13T11:32:40.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119077+0000", 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3647000000000003e-05 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:218 3.1 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.563498127000001, 2024-09-13T11:32:40.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036660768000000003, 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 218 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110588+0000", 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.761e-06 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110603+0000", 2024-09-13T11:32:40.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4542000000000001e-05 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110616+0000", 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2795e-05 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110628+0000", 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1959e-05 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114882+0000", 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042543379999999999 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000143676 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.120883+0000", 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058572549999999996 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.123587+0000", 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027042709999999998 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144002+0000", 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020414505999999999 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147169+0000", 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031675470000000002 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147204+0000", 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4523999999999997e-05 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147247+0000", 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3594000000000003e-05 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:219 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.563033771000001, 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.054051957999999997, 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 219 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.723 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111052+0000", 2024-09-13T11:32:40.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.093e-06 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111059+0000", 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9410000000000001e-06 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111069+0000", 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0519999999999999e-05 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111077+0000", 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8290000000000008e-06 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115373+0000", 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042952490000000001 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115459+0000", 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6367999999999997e-05 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.129610+0000", 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014151285 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.135388+0000", 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057775270000000002 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149276+0000", 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013888773 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165015+0000", 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015738682 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165065+0000", 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0170000000000002e-05 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165103+0000", 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7521000000000002e-05 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.728 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:221 3.1 3:baf08732:::smithi20438004-22 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.557641, 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034943881000000003, 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.729 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 221 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116445+0000", 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1680000000000001e-06 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116451+0000", 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0379999999999996e-06 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116461+0000", 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0604e-05 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116469+0000", 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8359999999999992e-06 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116516+0000", 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6332999999999998e-05 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116579+0000", 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3825999999999995e-05 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127789+0000", 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011209869000000001 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132875+0000", 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050855780000000003 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149082+0000", 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016207213000000002 2024-09-13T11:32:40.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151312+0000", 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002230358 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151344+0000", 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1081000000000003e-05 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151388+0000", 2024-09-13T11:32:40.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3977000000000003e-05 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:230 3.1 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.527490526999999, 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046965185999999999, 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.735 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146596+0000", 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.717e-06 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146606+0000", 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146620+0000", 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4343e-05 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146630+0000", 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3989999999999993e-06 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146705+0000", 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5135999999999993e-05 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146828+0000", 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122714 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165695+0000", 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018866951 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.171485+0000", 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057905220000000002 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185089+0000", 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013604132999999999 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193461+0000", 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083717840000000002 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193507+0000", 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5652000000000003e-05 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193559+0000", 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2624000000000003e-05 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:231 3.1 3:ab075b6a:::smithi20438004-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.527281176000001, 2024-09-13T11:32:40.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033558088, 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 231 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:32:40.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146804+0000", 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3300000000000002e-07 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146808+0000", 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4790000000000001e-06 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146814+0000", 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3230000000000003e-06 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146817+0000", 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4769999999999999e-06 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146955+0000", 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013720299999999999 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147052+0000", 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7231000000000005e-05 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.156769+0000", 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097173290000000002 2024-09-13T11:32:40.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165122+0000", 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083524589999999996 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.173409+0000", 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082871379999999994 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180256+0000", 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006846871 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180302+0000", 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6545999999999998e-05 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180362+0000", 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9499000000000001e-05 2024-09-13T11:32:40.747 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:235 3.1 3:be4a0686:::smithi20438004-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.524282749999999, 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035212407000000001, 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.748 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 235 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149803+0000", 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0580000000000001e-06 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149810+0000", 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149820+0000", 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0464e-05 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149826+0000", 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149863+0000", 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7416999999999998e-05 2024-09-13T11:32:40.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149983+0000", 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011927800000000001 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.160853+0000", 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010870216 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.167925+0000", 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070717159999999996 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180224+0000", 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012298972 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184892+0000", 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046685839999999999 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184959+0000", 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7105000000000002e-05 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185014+0000", 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5099000000000002e-05 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:236 3.1 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.754 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.524168834999999, 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059325392999999997, 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 236 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6599999999999996e-07 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149920+0000", 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149929+0000", 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3769999999999996e-06 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149937+0000", 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150123+0000", 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018567200000000001 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150254+0000", 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130426 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181070+0000", 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030816856 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.190793+0000", 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097220870000000008 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197985+0000", 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007192431 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209174+0000", 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011189536999999999 2024-09-13T11:32:40.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209209+0000", 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4359000000000001e-05 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209241+0000", 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2369000000000001e-05 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:237 3.1 3:9a2730a7:::smithi20438004-38 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.522216048000001, 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052738543999999998, 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 237 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151870+0000", 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151875+0000", 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2859999999999999e-06 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151890+0000", 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5557999999999999e-05 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151898+0000", 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7500000000000003e-06 2024-09-13T11:32:40.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151934+0000", 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5988000000000002e-05 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151999+0000", 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4394999999999996e-05 2024-09-13T11:32:40.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.177811+0000", 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025812831000000001 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185336+0000", 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075243209999999996 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197387+0000", 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012051345999999999 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204491+0000", 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071034999999999996 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204551+0000", 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0003000000000001e-05 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204607+0000", 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:245 3.1 3:adf25eb4:::smithi20438004-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.493027, 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.03651658, 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.767 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 245 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181059+0000", 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181066+0000", 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5119999999999997e-06 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181078+0000", 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2333e-05 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181085+0000", 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4449999999999999e-06 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185114+0000", 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040285579999999998 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1935000000000007e-05 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.195343+0000", 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010137147000000001 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200849+0000", 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005505874 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.203835+0000", 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002985705 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217494+0000", 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013658926 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217538+0000", 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4048999999999998e-05 2024-09-13T11:32:40.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217574+0000", 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6618000000000001e-05 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:249 3.1 3:81faf980:::smithi20438004-50 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.480108351, 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038339094999999997, 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.773 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 249 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193977+0000", 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.11e-06 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193984+0000", 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0794e-05 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194002+0000", 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.374e-06 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194061+0000", 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9485999999999999e-05 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194291+0000", 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000229321 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222511+0000", 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028219767999999999 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.227779+0000", 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052686319999999997 2024-09-13T11:32:40.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.231941+0000", 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041617809999999998 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232226+0000", 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028545799999999999 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232277+0000", 2024-09-13T11:32:40.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0445000000000002e-05 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232315+0000", 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584999999999997e-05 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:1 3.1 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.200047+0000", 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.474037697, 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017678494999999999, 2024-09-13T11:32:40.779 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200047+0000", 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200047+0000", 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200049+0000", 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5999999999999999e-06 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200054+0000", 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5799999999999999e-06 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200067+0000", 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2798e-05 2024-09-13T11:32:40.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200077+0000", 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0166000000000001e-05 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200129+0000", 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1468000000000003e-05 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200241+0000", 2024-09-13T11:32:40.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111919 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200452+0000", 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000211416 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201103+0000", 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00065099099999999996 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204160+0000", 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030574479999999999 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217615+0000", 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013454848 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217671+0000", 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5513999999999997e-05 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217725+0000", 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4747000000000003e-05 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.11.21:2 3.1 3:ab075b6a:::smithi20438004-32 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.239657+0000", 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.434427285, 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081779850000000005, 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.785 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239657+0000", 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239657+0000", 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239658+0000", 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0890000000000001e-06 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239664+0000", 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1309999999999997e-06 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239673+0000", 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5810000000000003e-06 2024-09-13T11:32:40.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239681+0000", 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8699999999999992e-06 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239721+0000", 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0046999999999998e-05 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239768+0000", 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7151999999999998e-05 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239873+0000", 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104728 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.240865+0000", 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00099215099999999993 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.244006+0000", 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003140772 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.247768+0000", 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037618339999999999 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.247811+0000", 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3130999999999998e-05 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.247835+0000", 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4499e-05 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.1.22:42 3.1 3:87527c9e:::cae462634b7a968897fe5e1960a361d42b8eb9292891e6b1d17dd542420f2e5f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.710200+0000", 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 12.963884338, 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088639080000000002, 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.791 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710200+0000", 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710200+0000", 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710201+0000", 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710313+0000", 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111728 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710320+0000", 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0779999999999998e-06 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710324+0000", 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8519999999999997e-06 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710352+0000", 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8169000000000001e-05 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.710355+0000", 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7870000000000002e-06 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.715819+0000", 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054640660000000001 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.715893+0000", 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4170000000000003e-05 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.716037+0000", 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014392499999999999 2024-09-13T11:32:40.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.717070+0000", 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001033541 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.717333+0000", 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026281500000000001 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.718985+0000", 2024-09-13T11:32:40.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001651961 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.719022+0000", 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.697e-05 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.719064+0000", 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2234999999999998e-05 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1382 3.1 3:871ca270:::d07c99508293d84e4ef6e64f0dc5690227be6f0a554acee36ad011709b311285:head [write 0~1144832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:36.084276+0000", 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.5898083200000004, 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093481090000000003, 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1382 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.798 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.084276+0000", 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.084276+0000", 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.084279+0000", 2024-09-13T11:32:40.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.154e-06 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.087857+0000", 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003578936 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.087876+0000", 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8281e-05 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.087884+0000", 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.721e-06 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.087962+0000", 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8047999999999995e-05 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.088114+0000", 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015170799999999999 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:40.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.088341+0000", 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022643000000000001 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.090502+0000", 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021614070000000002 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.093178+0000", 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026759029999999999 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.093560+0000", 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038168300000000001 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.093592+0000", 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2254e-05 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.093624+0000", 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2583999999999998e-05 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.803 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.804 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:40.804 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:40.804 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:40.805 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:40.804+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:40.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:32:40.840 INFO:tasks.rados.rados.0.smithi204.stdout:873: finishing set_chunk to oid 4 2024-09-13T11:32:40.840 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 674816 length: 17408 2024-09-13T11:32:40.840 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:874: finishing set_chunk to oid 9 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 32 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:875: finishing write tid 1 to smithi20438004-38 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:870: done (9 left) 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:871: done (8 left) 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:873: done (7 left) 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:874: done (6 left) 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:880: done (5 left) 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:881: write (excl) oid 30 current snap is 0 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:875: finishing write tid 2 to smithi20438004-38 2024-09-13T11:32:40.841 INFO:tasks.rados.rados.0.smithi204.stdout:876: expect (ObjNum 79 snap 0 seq_num 79) 2024-09-13T11:32:40.906 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:40.904+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:40.909 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:40.910 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:40.910 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:40.910 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:40.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:32:40.946 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 696132 2024-09-13T11:32:40.988 INFO:tasks.rados.rados.0.smithi204.stdout:875: finishing write tid 3 to smithi20438004-38 2024-09-13T11:32:40.988 INFO:tasks.rados.rados.0.smithi204.stdout:875: finishing write tid 4 to smithi20438004-38 2024-09-13T11:32:40.988 INFO:tasks.rados.rados.0.smithi204.stdout:875: oid 38 updating version 0 to 56 2024-09-13T11:32:40.988 INFO:tasks.rados.rados.0.smithi204.stdout:875: oid 38 updating version 56 to 58 2024-09-13T11:32:40.988 INFO:tasks.rados.rados.0.smithi204.stdout:875: oid 38 version 58 is already newer than 57 2024-09-13T11:32:40.989 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 58 (ObjNum 130 snap 0 seq_num 130) dirty exists 2024-09-13T11:32:40.989 INFO:tasks.rados.rados.0.smithi204.stdout:875: left oid 38 (ObjNum 130 snap 0 seq_num 130) 2024-09-13T11:32:40.989 INFO:tasks.rados.rados.0.smithi204.stdout:881: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-13T11:32:40.994 INFO:tasks.rados.rados.0.smithi204.stdout:881: writing smithi20438004-30 from 652138 to 1113505 tid 1 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:881: writing smithi20438004-30 from 1574940 to 2252138 tid 2 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:875: done (5 left) 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:876: done (4 left) 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:882: read oid 34 snap -1 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:882: expect (ObjNum 124 snap 0 seq_num 124) 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:883: read oid 44 snap -1 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:883: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:884: read oid 22 snap -1 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:884: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:885: read oid 45 snap -1 2024-09-13T11:32:41.001 INFO:tasks.rados.rados.0.smithi204.stdout:885: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-13T11:32:41.002 INFO:tasks.rados.rados.0.smithi204.stdout:886: set_chunk oid 5 target oid 2024-09-13T11:32:41.008 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:41.006+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:41.023 INFO:tasks.rados.rados.0.smithi204.stdout:886: set_chunk oid 5 offset: 419328 length: 1568256 target oid 237bea86f9e178a6b1be058269e0ed38743d02e4eeb67599426e8c5eda6345fb offset: 0 snap -1 2024-09-13T11:32:41.023 INFO:tasks.rados.rados.0.smithi204.stdout:887: set_chunk oid 47 target oid 2024-09-13T11:32:41.028 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:41.028 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:41.028 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:41.028 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:41.029 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:41.032 INFO:tasks.rados.rados.0.smithi204.stdout:887: set_chunk oid 47 offset: 687104 length: 347648 target oid 7fe204b5215839ac8b3466a1433f06bbdf4b446ecc5d338da23554be23fa9805 offset: 0 snap -1 2024-09-13T11:32:41.032 INFO:tasks.rados.rados.0.smithi204.stdout:888: read oid 43 snap -1 2024-09-13T11:32:41.032 INFO:tasks.rados.rados.0.smithi204.stdout:888: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:41.032 INFO:tasks.rados.rados.0.smithi204.stdout:889: set_chunk oid 2 target oid 2024-09-13T11:32:41.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:32:41.039 INFO:tasks.rados.rados.0.smithi204.stdout:889: set_chunk oid 2 offset: 63488 length: 382976 target oid c139272303a666242c217c459749c765a1ebb635be13fa231ffa3c00c20ddc0a offset: 0 snap -1 2024-09-13T11:32:41.040 INFO:tasks.rados.rados.0.smithi204.stdout:890: set_chunk oid 13 target oid 2024-09-13T11:32:41.042 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [2] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:41.043 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:32:41.043 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:32:41.043 INFO:tasks.thrashosds.thrasher:Checking the number of in OSDs in host ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:32:41.043 INFO:tasks.thrashosds.thrasher:Removing all OSDs in host ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:32:41.043 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-13T11:32:41.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-13T11:32:41.072 INFO:tasks.rados.rados.0.smithi204.stdout:890: set_chunk oid 13 offset: 510464 length: 2621952 target oid 2c9d97cd8bc343ec27e96488c4591170a3e62f503f5d642c2598a4d2866780e1 offset: 0 snap -1 2024-09-13T11:32:41.072 INFO:tasks.rados.rados.0.smithi204.stdout:891: read oid 37 snap -1 2024-09-13T11:32:41.072 INFO:tasks.rados.rados.0.smithi204.stdout:891: expect (ObjNum 114 snap 0 seq_num 114) 2024-09-13T11:32:41.072 INFO:tasks.rados.rados.0.smithi204.stdout:892: write oid 24 current snap is 0 2024-09-13T11:32:41.072 INFO:tasks.rados.rados.0.smithi204.stdout:877: finishing write tid 1 to smithi20438004-7 2024-09-13T11:32:41.072 INFO:tasks.rados.rados.0.smithi204.stdout:892: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-13T11:32:41.078 INFO:tasks.rados.rados.0.smithi204.stdout:892: writing smithi20438004-24 from 668945 to 1205561 tid 1 2024-09-13T11:32:41.084 INFO:tasks.rados.rados.0.smithi204.stdout:892: writing smithi20438004-24 from 1633422 to 2124785 tid 2 2024-09-13T11:32:41.084 INFO:tasks.rados.rados.0.smithi204.stdout:893: read oid 42 snap -1 2024-09-13T11:32:41.084 INFO:tasks.rados.rados.0.smithi204.stdout:893: expect (ObjNum 126 snap 0 seq_num 126) 2024-09-13T11:32:41.084 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:41.084 INFO:tasks.rados.rados.0.smithi204.stdout:877: finishing write tid 2 to smithi20438004-7 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:877: finishing write tid 3 to smithi20438004-7 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:878: done (15 left) 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:894: write oid 26 current snap is 0 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:877: finishing write tid 4 to smithi20438004-7 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:877: oid 7 updating version 0 to 36 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:877: oid 7 updating version 36 to 38 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:877: oid 7 version 38 is already newer than 37 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 38 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:877: left oid 7 (ObjNum 131 snap 0 seq_num 131) 2024-09-13T11:32:41.085 INFO:tasks.rados.rados.0.smithi204.stdout:879: finishing write tid 1 to smithi20438004-28 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:879: finishing write tid 2 to smithi20438004-28 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:879: finishing write tid 3 to smithi20438004-28 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:879: finishing write tid 4 to smithi20438004-28 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:879: oid 28 updating version 0 to 43 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:879: oid 28 version 43 is already newer than 41 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:879: oid 28 version 43 is already newer than 42 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 43 (ObjNum 132 snap 0 seq_num 132) dirty exists 2024-09-13T11:32:41.086 INFO:tasks.rados.rados.0.smithi204.stdout:879: left oid 28 (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:41.087 INFO:tasks.rados.rados.0.smithi204.stdout:881: finishing write tid 1 to smithi20438004-30 2024-09-13T11:32:41.087 INFO:tasks.rados.rados.0.smithi204.stdout:894: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-13T11:32:41.090 INFO:tasks.rados.rados.0.smithi204.stdout:894: writing smithi20438004-26 from 685752 to 1297617 tid 1 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout:894: writing smithi20438004-26 from 1975551 to 2285752 tid 2 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout:877: done (15 left) 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout:879: done (14 left) 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout:895: read oid 49 snap -1 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout:895: expect deleted 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout:896: read oid 12 snap -1 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout:896: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-13T11:32:41.097 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:881: finishing write tid 2 to smithi20438004-30 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:881: finishing write tid 3 to smithi20438004-30 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:881: finishing write tid 4 to smithi20438004-30 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:881: oid 30 updating version 0 to 47 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:881: oid 30 updating version 47 to 49 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:881: oid 30 version 49 is already newer than 48 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 49 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-13T11:32:41.098 INFO:tasks.rados.rados.0.smithi204.stdout:881: left oid 30 (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:32:41.099 INFO:tasks.rados.rados.0.smithi204.stdout:883: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:41.099 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 161508 2024-09-13T11:32:41.109 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:41.107+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:41.125 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 465130 2024-09-13T11:32:41.144 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:41.144 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:7 4.6 e38/31 4:6d3e8486:::smithi20438004-2:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.473060+0000", 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.668449238000001, 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057936189999999999, 2024-09-13T11:32:41.145 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473060+0000", 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473060+0000", 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473060+0000", 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473069+0000", 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1230000000000006e-06 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473076+0000", 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2309999999999999e-06 2024-09-13T11:32:41.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473082+0000", 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5269999999999998e-06 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473519+0000", 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043691800000000001 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.473535+0000", 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6239000000000002e-05 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.478816+0000", 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052805309999999998 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.478853+0000", 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7490000000000002e-05 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.149 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:14 4.6 e38/31 4:600f3d3f:::smithi20438004-4:head v 38'5, mlcod=38'3)", 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.495049+0000", 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.646459598, 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061095400000000001, 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495049+0000", 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495049+0000", 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495050+0000", 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2200000000000003e-07 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495393+0000", 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034316000000000001 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495405+0000", 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2267e-05 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495411+0000", 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2950000000000001e-06 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495459+0000", 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8114e-05 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495478+0000", 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9313999999999998e-05 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.501109+0000", 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056309469999999999 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.501159+0000", 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9721e-05 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.155 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:15 4.6 e38/31 4:600f3d3f:::smithi20438004-4:head v 38'6, mlcod=38'3)", 2024-09-13T11:32:41.155 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.495545+0000", 2024-09-13T11:32:41.155 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.645963323, 2024-09-13T11:32:41.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056420910000000001, 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495545+0000", 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495545+0000", 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495546+0000", 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6199999999999998e-07 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495551+0000", 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1370000000000001e-06 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495557+0000", 2024-09-13T11:32:41.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9660000000000001e-06 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.495562+0000", 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9830000000000004e-06 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.496043+0000", 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048044899999999998 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.496058+0000", 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.52e-05 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.501164+0000", 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051062970000000001 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.501188+0000", 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3496999999999999e-05 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:73 4.f 4:f653a832:::smithi20438004-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.669206+0000", 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.472302513000001, 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066877550000000001, 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 73 2024-09-13T11:32:41.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.669206+0000", 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.669206+0000", 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.669208+0000", 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.609e-06 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670000+0000", 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00079190900000000004 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670015+0000", 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5026e-05 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670022+0000", 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8900000000000001e-06 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670136+0000", 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011458199999999999 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670250+0000", 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011354 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670346+0000", 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5889000000000003e-05 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.672746+0000", 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002399862 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675787+0000", 2024-09-13T11:32:41.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030412740000000001 2024-09-13T11:32:41.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675804+0000", 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7119999999999999e-05 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675842+0000", 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8507999999999998e-05 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675894+0000", 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1545999999999997e-05 2024-09-13T11:32:41.166 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.13.23:11 4.f 4:f5aacf61:::smithi20438004-27:head [assert-version v9,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.134856+0000", 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.006652434999999, 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059802400000000004, 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.134856+0000", 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.134856+0000", 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.134857+0000", 2024-09-13T11:32:41.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2799999999999996e-07 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.134861+0000", 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5199999999999999e-06 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.134867+0000", 2024-09-13T11:32:41.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8300000000000001e-06 2024-09-13T11:32:41.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.134871+0000", 2024-09-13T11:32:41.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4860000000000002e-06 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.138668+0000", 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037969969999999999 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.138721+0000", 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3038000000000003e-05 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140837+0000", 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021158409999999998 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.6.22:12 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.150118+0000", 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.991390672, 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055169520000000003, 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150118+0000", 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150118+0000", 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150120+0000", 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2010000000000002e-06 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150132+0000", 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1868999999999999e-05 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150153+0000", 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0981000000000001e-05 2024-09-13T11:32:41.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150164+0000", 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150236+0000", 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2849000000000007e-05 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150332+0000", 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6001000000000005e-05 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.155635+0000", 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053027029999999998 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.176 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:254 4.0 4:04aeeab9:::smithi20438004-5:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.939916030999999, 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022108323999999999, 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 254 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201594+0000", 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201603+0000", 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.568e-06 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201619+0000", 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6540000000000001e-05 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201629+0000", 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3200000000000006e-06 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201685+0000", 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6078000000000001e-05 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201747+0000", 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2242999999999993e-05 2024-09-13T11:32:41.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220103+0000", 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01835548 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220937+0000", 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083429299999999995 2024-09-13T11:32:41.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221938+0000", 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010010710000000001 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223567+0000", 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001628846 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223638+0000", 2024-09-13T11:32:41.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1165000000000001e-05 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223701+0000", 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3120999999999994e-05 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:268 4.f 4:f653a832:::smithi20438004-19:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.183 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:32:41.184 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.917550624, 2024-09-13T11:32:41.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01043647, 2024-09-13T11:32:41.184 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.184 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.184 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.184 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 268 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223960+0000", 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.641e-06 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223968+0000", 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1820000000000006e-06 2024-09-13T11:32:41.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223982+0000", 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3671e-05 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223989+0000", 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224062+0000", 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2315000000000001e-05 2024-09-13T11:32:41.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224166+0000", 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010401299999999999 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232681+0000", 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085151910000000001 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233615+0000", 2024-09-13T11:32:41.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00093373600000000001 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234090+0000", 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000475699 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234332+0000", 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241895 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234362+0000", 2024-09-13T11:32:41.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9889000000000001e-05 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234395+0000", 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2471000000000001e-05 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.190 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:274 4.f 4:f3c80275:::smithi20438004-25:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.228974+0000", 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.912534301000001, 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010225435999999999, 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 274 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.191 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228974+0000", 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228974+0000", 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228976+0000", 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4729999999999999e-06 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228983+0000", 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5229999999999998e-06 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229000+0000", 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6748e-05 2024-09-13T11:32:41.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229009+0000", 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8480000000000007e-06 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229077+0000", 2024-09-13T11:32:41.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7637000000000003e-05 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229172+0000", 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5486e-05 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233860+0000", 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046882069999999998 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234569+0000", 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00070882900000000001 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.235208+0000", 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00063829299999999996 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239093+0000", 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038850719999999998 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239148+0000", 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5562000000000001e-05 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239200+0000", 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1758000000000001e-05 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:276 4.f 4:f5aacf61:::smithi20438004-27:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.230800+0000", 2024-09-13T11:32:41.197 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.910709238000001, 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084841529999999995, 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 276 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230800+0000", 2024-09-13T11:32:41.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230800+0000", 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230801+0000", 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5300000000000002e-07 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230808+0000", 2024-09-13T11:32:41.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7570000000000004e-06 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230819+0000", 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0266e-05 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230825+0000", 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6150000000000002e-06 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230873+0000", 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7738000000000003e-05 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230930+0000", 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6879000000000002e-05 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234840+0000", 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039098249999999996 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.235519+0000", 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00067980500000000001 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.235667+0000", 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014805100000000001 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239208+0000", 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003540975 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239250+0000", 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1103999999999998e-05 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239284+0000", 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4184999999999999e-05 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.203 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:277 4.0 4:001d90c2:::smithi20438004-28:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.230840+0000", 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.910669105, 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0087174109999999996, 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.204 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.205 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 277 2024-09-13T11:32:41.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.205 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230840+0000", 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230840+0000", 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230840+0000", 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7999999999999996e-07 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230843+0000", 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2590000000000001e-06 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230849+0000", 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5990000000000001e-06 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230859+0000", 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0369e-05 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.230918+0000", 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8677000000000001e-05 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.231000+0000", 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1794999999999999e-05 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.235232+0000", 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042325009999999996 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236034+0000", 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080161900000000005 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239457+0000", 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034229939999999999 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239490+0000", 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2926999999999998e-05 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239524+0000", 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3911000000000001e-05 2024-09-13T11:32:41.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239557+0000", 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.328e-05 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:285 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.906931572, 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012191677, 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 285 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234578+0000", 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234583+0000", 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9110000000000001e-06 2024-09-13T11:32:41.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234593+0000", 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234598+0000", 2024-09-13T11:32:41.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9509999999999997e-06 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234639+0000", 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1826999999999998e-05 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234688+0000", 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9039000000000003e-05 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:32:41.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239203+0000", 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045143739999999998 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239979+0000", 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077664100000000001 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.242634+0000", 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026544899999999998 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246683+0000", 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040491499999999996 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246723+0000", 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0429000000000002e-05 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246769+0000", 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5430999999999999e-05 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:288 4.f 4:fea8c5b2:::smithi20438004-39:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.236775+0000", 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.904733974999999, 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01003386, 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 288 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.219 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236775+0000", 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236775+0000", 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236776+0000", 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4700000000000001e-07 2024-09-13T11:32:41.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236780+0000", 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0409999999999999e-06 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236790+0000", 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0079e-05 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236795+0000", 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2070000000000003e-06 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236839+0000", 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4156000000000002e-05 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.236885+0000", 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6176e-05 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.240343+0000", 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034573360000000001 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.241153+0000", 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081013199999999997 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.241888+0000", 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073577799999999997 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246713+0000", 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048240890000000002 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246766+0000", 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3368999999999997e-05 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246809+0000", 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2750000000000002e-05 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:577 4.0 4:001d90c2:::smithi20438004-28:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.306576+0000", 2024-09-13T11:32:41.225 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.834932713000001, 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0087902459999999998, 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 577 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306576+0000", 2024-09-13T11:32:41.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306576+0000", 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306577+0000", 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3099999999999996e-07 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306581+0000", 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4170000000000002e-06 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306593+0000", 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.134e-05 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306600+0000", 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8900000000000001e-06 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306655+0000", 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5254000000000003e-05 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.306702+0000", 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6903999999999997e-05 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.313800+0000", 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070983569999999996 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.314234+0000", 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000434394 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.314787+0000", 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055216200000000003 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.315303+0000", 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00051685299999999995 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.315336+0000", 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2225999999999999e-05 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.315366+0000", 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0617999999999997e-05 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.232 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1053 4.0 4:04aeeab9:::smithi20438004-5:head [read 0~3075648,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:30.753612+0000", 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.387897053, 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054775650000000002, 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1053 2024-09-13T11:32:41.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753612+0000", 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753612+0000", 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753612+0000", 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5900000000000002e-07 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753615+0000", 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8030000000000001e-06 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753621+0000", 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753625+0000", 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5310000000000003e-06 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753711+0000", 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6388e-05 2024-09-13T11:32:41.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.753716+0000", 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6040000000000003e-06 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.758100+0000", 2024-09-13T11:32:41.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043836789999999997 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.758151+0000", 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0985000000000001e-05 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.759089+0000", 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00093857200000000004 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.238 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1059 4.f 4:fea8c5b2:::smithi20438004-39:head [write 594981~609197] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:30.778137+0000", 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.363371495999999, 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059679859999999998, 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.239 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1059 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778137+0000", 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778137+0000", 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778140+0000", 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8370000000000001e-06 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778686+0000", 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054575000000000001 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778699+0000", 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3434999999999999e-05 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778707+0000", 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3719999999999999e-06 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778753+0000", 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6165999999999998e-05 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778813+0000", 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0130000000000002e-05 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.778908+0000", 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4629999999999994e-05 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.783597+0000", 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046889030000000003 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.783922+0000", 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032514499999999999 2024-09-13T11:32:41.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.784007+0000", 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4963000000000006e-05 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.784065+0000", 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8325999999999997e-05 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.784105+0000", 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0329e-05 2024-09-13T11:32:41.244 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1148 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~3394981,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:32.057280+0000", 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.0842292439999994, 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082169229999999992, 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1148 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057280+0000", 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057280+0000", 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057280+0000", 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4999999999999998e-07 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057283+0000", 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6079999999999998e-06 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057289+0000", 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.601e-06 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057294+0000", 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1100000000000002e-06 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.061476+0000", 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004181756 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.061526+0000", 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9506999999999999e-05 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.065496+0000", 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039708909999999998 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.249 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1149 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~3394981,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:32.057301+0000", 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.0842081649999997, 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099935700000000002, 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.250 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1149 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057301+0000", 2024-09-13T11:32:41.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057301+0000", 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057301+0000", 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3099999999999998e-07 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057304+0000", 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.813e-06 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057309+0000", 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6700000000000002e-06 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.057313+0000", 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.746e-06 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.065548+0000", 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082351219999999992 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.065632+0000", 2024-09-13T11:32:41.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4016000000000003e-05 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:32.067294+0000", 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016617719999999999 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1423 4.0 4:04aeeab9:::smithi20438004-5:head [write 2919420~616377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:41.255 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:36.775506+0000", 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.3660024269999997, 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057931040000000003, 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1423 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.775506+0000", 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.775506+0000", 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.775507+0000", 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1060000000000001e-06 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.776043+0000", 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053514299999999997 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.776053+0000", 2024-09-13T11:32:41.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0380000000000001e-05 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.776060+0000", 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7839999999999998e-06 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.776140+0000", 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0344000000000002e-05 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.776188+0000", 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7889999999999997e-05 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.776267+0000", 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8537999999999999e-05 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.779060+0000", 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027937180000000002 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.780156+0000", 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001095408 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.781252+0000", 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001095822 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.781278+0000", 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6081e-05 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.781299+0000", 2024-09-13T11:32:41.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1889999999999999e-05 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1589 4.f 4:f3c80275:::smithi20438004-25:head [write 568103~768769] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:40.305075+0000", 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.83643396000000003, 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069647270000000004, 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.263 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1589 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305075+0000", 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305075+0000", 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305076+0000", 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.003e-06 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305774+0000", 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000697797 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305788+0000", 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4825e-05 2024-09-13T11:32:41.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305796+0000", 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1130000000000003e-06 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305844+0000", 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8673e-05 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.305911+0000", 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7149000000000004e-05 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.306011+0000", 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9724999999999998e-05 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.308597+0000", 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025859490000000001 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.311880+0000", 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032833319999999999 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.311925+0000", 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4119999999999998e-05 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.311981+0000", 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6091999999999998e-05 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:40.312040+0000", 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8949000000000001e-05 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.270 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:41.271 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.272 INFO:tasks.rados.rados.0.smithi204.stdout:885: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-13T11:32:41.272 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:32:41.273 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:41.270+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:41.320 INFO:tasks.rados.rados.0.smithi204.stdout:882: expect (ObjNum 124 snap 0 seq_num 124) 2024-09-13T11:32:41.349 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 362060 2024-09-13T11:32:41.358 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:32:41.357+0000 7f8352d34640 -1 mon.a@0(leader).osd e45 definitely_dead 0 2024-09-13T11:32:41.373 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:41.371+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:41.376 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:41.376 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:41.377 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:32:41.377 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:41.378 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:32:41.410 INFO:tasks.rados.rados.0.smithi204.stdout:886: finishing set_chunk to oid 5 2024-09-13T11:32:41.410 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 40 (ObjNum 128 snap 0 seq_num 128) dirty exists 2024-09-13T11:32:41.410 INFO:tasks.rados.rados.0.smithi204.stdout:884: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-13T11:32:41.443 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.473 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:41.471+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:41.488 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:41.488 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:41.488 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:41.488 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:41.488 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:41.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:32:41.520 INFO:tasks.rados.rados.0.smithi204.stdout:889: finishing set_chunk to oid 2 2024-09-13T11:32:41.520 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-13T11:32:41.520 INFO:tasks.rados.rados.0.smithi204.stdout:887: finishing set_chunk to oid 47 2024-09-13T11:32:41.520 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 31 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-13T11:32:41.520 INFO:tasks.rados.rados.0.smithi204.stdout:888: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:41.520 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.525 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.1. 2024-09-13T11:32:41.541 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-13T11:32:41.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-09-13T11:32:41.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:32:41.582 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:41.581+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:41.598 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1 4.e e38/31 4:76a15ccf:::smithi20438004-1:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.456418+0000", 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 15.143756023, 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046371939999999999, 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.602 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.456418+0000", 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.456418+0000", 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.456419+0000", 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5000000000000003e-07 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457207+0000", 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078827599999999997 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457220+0000", 2024-09-13T11:32:41.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3134e-05 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457230+0000", 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0084e-05 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457283+0000", 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2948000000000002e-05 2024-09-13T11:32:41.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457302+0000", 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8675e-05 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.460991+0000", 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036891490000000001 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.606 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.461055+0000", 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4377999999999999e-05 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:9 4.9 e38/31 4:94e25a0d:::smithi20438004-3:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.486052+0000", 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 15.114122386, 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028537229999999999, 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.607 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.486052+0000", 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.486052+0000", 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.486052+0000", 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8000000000000002e-07 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.487286+0000", 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001233966 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.487300+0000", 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4005999999999999e-05 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.487307+0000", 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4320000000000002e-06 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.487367+0000", 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9599000000000003e-05 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.487388+0000", 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0806999999999999e-05 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488878+0000", 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001489857 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488905+0000", 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7776e-05 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:11 4.9 e38/31 4:94e25a0d:::smithi20438004-3:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.488314+0000", 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 15.111859723, 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030730190000000002, 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488314+0000", 2024-09-13T11:32:41.612 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488314+0000", 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488315+0000", 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.99e-07 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488320+0000", 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5990000000000004e-06 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488325+0000", 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8000000000000004e-06 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488344+0000", 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8966000000000001e-05 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488756+0000", 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041219199999999997 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.488771+0000", 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4301000000000001e-05 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.491340+0000", 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025691429999999999 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.491387+0000", 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7419000000000001e-05 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:21 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'4, mlcod=38'2)", 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.513425+0000", 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 15.086748739000001, 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030449259999999999, 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.616 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.513425+0000", 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.513425+0000", 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.513427+0000", 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2440000000000001e-06 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.514518+0000", 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010912490000000001 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.514533+0000", 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4904e-05 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.514541+0000", 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4570000000000005e-06 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.514593+0000", 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1480999999999997e-05 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.514611+0000", 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8181000000000001e-05 2024-09-13T11:32:41.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.516426+0000", 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001815604 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.516470+0000", 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3806e-05 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:73 4.f e38/31 4:f653a832:::smithi20438004-19:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.670905+0000", 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.929268719, 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004241224, 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670905+0000", 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670905+0000", 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670906+0000", 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2399999999999999e-07 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671267+0000", 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00036090800000000001 2024-09-13T11:32:41.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671278+0000", 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1188e-05 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671285+0000", 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671340+0000", 2024-09-13T11:32:41.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4877999999999999e-05 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671354+0000", 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4095e-05 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675095+0000", 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037409019999999999 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675147+0000", 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1616999999999997e-05 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:106 4.f e38/31 4:f5aacf61:::smithi20438004-27:head v 38'8, mlcod=38'6)", 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.774680+0000", 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.825494471000001, 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003619007, 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.625 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774680+0000", 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774680+0000", 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774680+0000", 2024-09-13T11:32:41.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5199999999999995e-07 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774860+0000", 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017915800000000001 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774872+0000", 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2317000000000001e-05 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774900+0000", 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7886999999999999e-05 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774955+0000", 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5612999999999998e-05 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.774976+0000", 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0724999999999999e-05 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.778256+0000", 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003279962 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.778299+0000", 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2493000000000001e-05 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:107 4.f e38/31 4:f5aacf61:::smithi20438004-27:head v 38'9, mlcod=38'6)", 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.775600+0000", 2024-09-13T11:32:41.629 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.824574288999999, 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028869149999999999, 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775600+0000", 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775600+0000", 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775600+0000", 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6000000000000003e-07 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775606+0000", 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5230000000000003e-06 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775616+0000", 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0179e-05 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775622+0000", 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0260000000000004e-06 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775671+0000", 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8548999999999999e-05 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.775686+0000", 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5767e-05 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.778460+0000", 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002773307 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.778487+0000", 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6903999999999999e-05 2024-09-13T11:32:41.633 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:154 4.f e38/31 4:fea8c5b2:::smithi20438004-39:head v 38'11, mlcod=38'9)", 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.929040+0000", 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.671134425, 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003707592, 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929040+0000", 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929040+0000", 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929040+0000", 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8599999999999997e-07 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929227+0000", 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000186489 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929239+0000", 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2414e-05 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929269+0000", 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0042000000000001e-05 2024-09-13T11:32:41.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929317+0000", 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8035999999999999e-05 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929342+0000", 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4726e-05 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.932708+0000", 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033653089999999999 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.932747+0000", 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9690000000000001e-05 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:155 4.f e38/31 4:fea8c5b2:::smithi20438004-39:head v 38'12, mlcod=38'9)", 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.929762+0000", 2024-09-13T11:32:41.638 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.670412410000001, 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030266310000000001, 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929762+0000", 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.639 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929762+0000", 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929762+0000", 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8999999999999995e-07 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929768+0000", 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7769999999999999e-06 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929779+0000", 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0363000000000001e-05 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929786+0000", 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0380000000000002e-06 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929934+0000", 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000148629 2024-09-13T11:32:41.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.929949+0000", 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4936000000000001e-05 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.932768+0000", 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028185350000000001 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.932788+0000", 2024-09-13T11:32:41.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0562999999999999e-05 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.578752201, 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005153202, 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021424+0000", 2024-09-13T11:32:41.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8759999999999999e-06 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022585+0000", 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011614539999999999 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022596+0000", 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1110999999999999e-05 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022603+0000", 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4250000000000003e-06 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022647+0000", 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4487999999999998e-05 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022662+0000", 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4817999999999999e-05 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.026542+0000", 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038797409999999999 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.026575+0000", 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3288999999999999e-05 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:182 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'8, mlcod=38'6)", 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.027632+0000", 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.572542275, 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023703700000000001, 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.027632+0000", 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.027632+0000", 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.027632+0000", 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3499999999999996e-07 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.028779+0000", 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00114624 2024-09-13T11:32:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.028787+0000", 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5990000000000002e-06 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.028792+0000", 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5909999999999998e-06 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.028857+0000", 2024-09-13T11:32:41.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4851999999999995e-05 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.028870+0000", 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3164e-05 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.029972+0000", 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011020190000000001 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.030002+0000", 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0369999999999999e-05 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:189 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'10, mlcod=38'8)", 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.050082+0000", 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.550092081000001, 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049158680000000003, 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.652 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050082+0000", 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050082+0000", 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050083+0000", 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9500000000000001e-07 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050921+0000", 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083774800000000003 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050932+0000", 2024-09-13T11:32:41.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.172e-05 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050958+0000", 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5252000000000001e-05 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.051034+0000", 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6756999999999997e-05 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.051060+0000", 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5868e-05 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.054961+0000", 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039008879999999999 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.054998+0000", 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6739999999999997e-05 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.656 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:190 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'11, mlcod=38'9)", 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.057440+0000", 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.542734423000001, 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023263810000000002, 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.057440+0000", 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.057440+0000", 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.057441+0000", 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4580000000000001e-06 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058508+0000", 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001066443 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058520+0000", 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2454e-05 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058526+0000", 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5050000000000004e-06 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058572+0000", 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5189999999999999e-05 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058588+0000", 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.668e-05 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.059734+0000", 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001145404 2024-09-13T11:32:41.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.059766+0000", 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2246999999999998e-05 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:191 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'12, mlcod=38'9)", 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.058543+0000", 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.541630705999999, 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034673669999999998, 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058543+0000", 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058543+0000", 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058544+0000", 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6899999999999998e-07 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058546+0000", 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6249999999999999e-06 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058551+0000", 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.279e-06 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.058554+0000", 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8e-06 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.059236+0000", 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00068191599999999997 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.059256+0000", 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9267999999999999e-05 2024-09-13T11:32:41.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.061983+0000", 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002727789 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.062011+0000", 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7321000000000002e-05 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.2053273, 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078075499999999999, 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394848+0000", 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397492+0000", 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026441139999999999 2024-09-13T11:32:41.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397513+0000", 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0381e-05 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397520+0000", 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4789999999999999e-06 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397586+0000", 2024-09-13T11:32:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5687999999999999e-05 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397612+0000", 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5870000000000001e-05 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402610+0000", 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049984199999999999 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402654+0000", 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4153000000000002e-05 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "pg_update_log_missing(4.e epoch 38/31 rep_tid 32 entries 38'23 (0'0) error 4:7022fc62:::smithi20438004-6:head by client.4631.0:787 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 38'20 roll_forward_to 38'20)", 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.200600393, 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057797359999999997, 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T11:32:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:32:41.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8500000000000001e-07 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399583+0000", 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4139999999999994e-06 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399602+0000", 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9176000000000001e-05 2024-09-13T11:32:41.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399611+0000", 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2520000000000005e-06 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399661+0000", 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9688000000000003e-05 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.405353+0000", 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056925209999999999 2024-09-13T11:32:41.673 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1060 4.f e40/31 4:fea8c5b2:::smithi20438004-39:head v 40'45, mlcod=39'43)", 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:30.787357+0000", 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.812816975, 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025361540000000001, 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787357+0000", 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787357+0000", 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787358+0000", 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:32:41.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787735+0000", 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037734900000000002 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787745+0000", 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0118e-05 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787751+0000", 2024-09-13T11:32:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2299999999999999e-06 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787796+0000", 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4879000000000002e-05 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.787811+0000", 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5119999999999999e-05 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.789854+0000", 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020431469999999999 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:30.789893+0000", 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9431000000000002e-05 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1439 4.f e44/31 4:f5aacf61:::smithi20438004-27:head v 44'59, mlcod=43'57)", 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:32:41.678 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.7874557299999996, 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039265439999999997, 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6299999999999999e-07 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813007+0000", 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028868900000000001 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813018+0000", 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0575e-05 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813026+0000", 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2169999999999994e-06 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813096+0000", 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0528e-05 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813115+0000", 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8508e-05 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.816613+0000", 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034979939999999999 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.816645+0000", 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1869999999999998e-05 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1440 4.f e44/31 4:f5aacf61:::smithi20438004-27:head v 44'60, mlcod=43'58)", 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:36.818815+0000", 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.7813589329999999, 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004036291, 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.818815+0000", 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.818815+0000", 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.818816+0000", 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8400000000000003e-07 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819080+0000", 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026429999999999997 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819090+0000", 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3989999999999993e-06 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819099+0000", 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0939999999999997e-06 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819211+0000", 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112755 2024-09-13T11:32:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819235+0000", 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3751e-05 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.822823+0000", 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035873300000000001 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.822851+0000", 2024-09-13T11:32:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8878e-05 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.568922208, 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081668139999999997, 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:32:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031253+0000", 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7900000000000007e-07 2024-09-13T11:32:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037708+0000", 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064553029999999999 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037720+0000", 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2062000000000001e-05 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037745+0000", 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4766999999999999e-05 2024-09-13T11:32:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037797+0000", 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2198999999999999e-05 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037815+0000", 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8178000000000001e-05 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039386+0000", 2024-09-13T11:32:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015708110000000001 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039419+0000", 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2515000000000003e-05 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:32:41.692 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:41.693 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.694 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:32:41.694 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:41.692+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:41.753 INFO:tasks.rados.rados.0.smithi204.stdout:890: finishing set_chunk to oid 13 2024-09-13T11:32:41.753 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 34 (ObjNum 117 snap 0 seq_num 117) dirty exists 2024-09-13T11:32:41.753 INFO:tasks.rados.rados.0.smithi204.stdout:891: expect (ObjNum 114 snap 0 seq_num 114) 2024-09-13T11:32:41.779 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1930872 2024-09-13T11:32:41.794 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:41.793+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:41.799 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:41.799 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:41.799 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:41.808 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:32:41.895 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:41.894+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:41.899 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:32:41.898+0000 7f8352d34640 -1 mon.a@0(leader).osd e46 definitely_dead 0 2024-09-13T11:32:41.912 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:41.912 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:41.912 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:41.920 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:32:41.954 INFO:tasks.rados.rados.0.smithi204.stdout:892: finishing write tid 1 to smithi20438004-24 2024-09-13T11:32:41.954 INFO:tasks.rados.rados.0.smithi204.stdout:893: expect (ObjNum 126 snap 0 seq_num 126) 2024-09-13T11:32:41.974 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:41.996 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:41.995+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:42.007 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 658658 2024-09-13T11:32:42.026 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:42.026 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:42.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:42.026 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'87, mlcod=24'81)", 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.952814609999997, 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0126724, 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071634+0000", 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9599999999999996e-05 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071640+0000", 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4980000000000003e-06 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071644+0000", 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2919999999999997e-06 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073407+0000", 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001762149 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073417+0000", 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0859999999999999e-05 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084190+0000", 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010772903 2024-09-13T11:32:42.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084216+0000", 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5622e-05 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:118 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'88, mlcod=24'81)", 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:42.031 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.952294596000002, 2024-09-13T11:32:42.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012784627999999999, 2024-09-13T11:32:42.033 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.033 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.033 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.033 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:42.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4900000000000001e-07 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072067+0000", 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2260000000000001e-06 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072074+0000", 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2820000000000002e-06 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072077+0000", 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5899999999999999e-06 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075773+0000", 2024-09-13T11:32:42.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003695884 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075786+0000", 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2627e-05 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084816+0000", 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090302000000000004 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084848+0000", 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2369999999999997e-05 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:122 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'92, mlcod=24'82)", 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.950341850000001, 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013166437, 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.039 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074017+0000", 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9500000000000003e-07 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074022+0000", 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0529999999999997e-06 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074029+0000", 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8419999999999999e-06 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074032+0000", 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8060000000000001e-06 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081498+0000", 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074659669999999996 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081510+0000", 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1265e-05 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087152+0000", 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056421889999999997 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087183+0000", 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.082e-05 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'82)", 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.949829757000003, 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012918696, 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074529+0000", 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8400000000000004e-07 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074601+0000", 2024-09-13T11:32:42.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2106000000000004e-05 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074611+0000", 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0054000000000001e-05 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074616+0000", 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6349999999999997e-06 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084250+0000", 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096340109999999996 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084263+0000", 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3292000000000001e-05 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087416+0000", 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003152672 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087447+0000", 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1241999999999997e-05 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'82)", 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.949304564999998, 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012417658, 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.048 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075054+0000", 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5600000000000001e-07 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075058+0000", 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6689999999999999e-06 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075065+0000", 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9269999999999999e-06 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075069+0000", 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.963e-06 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084645+0000", 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095762209999999993 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084657+0000", 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2017e-05 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087449+0000", 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002792189 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087471+0000", 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2215999999999999e-05 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.670046104000001, 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.053 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:32:42.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:42.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:209 3.2 e38/30 3:6afe79ef:::smithi20438004-10 low_tier:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.931515096, 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016793596000000001, 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:42.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4799999999999999e-07 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102771+0000", 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0099271120000000001 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102791+0000", 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0148000000000001e-05 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102801+0000", 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8560000000000002e-06 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102898+0000", 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7528000000000005e-05 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102915+0000", 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6915999999999999e-05 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109590+0000", 2024-09-13T11:32:42.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066751579999999996 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109637+0000", 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6529999999999997e-05 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.921515183, 2024-09-13T11:32:42.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.909297588999999, 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:32:42.067 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:32:42.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:32:42.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:207 3.2 e38/30 3:59a396eb:::smithi20438004-8 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.902547885000001, 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025976583000000001, 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121811+0000", 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2799999999999996e-07 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128273+0000", 2024-09-13T11:32:42.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064622129999999996 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128294+0000", 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1364000000000001e-05 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128312+0000", 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8002000000000001e-05 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128443+0000", 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013052699999999999 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128487+0000", 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4338999999999997e-05 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147644+0000", 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019156341 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147787+0000", 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014326900000000001 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:220 3.2 e38/30 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.888880255, 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013667876000000001, 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.076 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.077 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135479+0000", 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.175e-06 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139134+0000", 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036550240000000002 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139152+0000", 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8136999999999999e-05 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139168+0000", 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5951e-05 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139254+0000", 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5691000000000003e-05 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139278+0000", 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4148e-05 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149117+0000", 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098384820000000008 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149146+0000", 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9268000000000002e-05 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:233 3.2 e38/30 3:53d0adee:::smithi20438004-34 low_tier:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:42.080 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.871399736000001, 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023237916000000001, 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152960+0000", 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161956+0000", 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0089955509999999992 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161973+0000", 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7350000000000002e-05 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161984+0000", 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.11e-05 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162047+0000", 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2677999999999999e-05 2024-09-13T11:32:42.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162075+0000", 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8501e-05 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176139+0000", 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014063512 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176196+0000", 2024-09-13T11:32:42.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7661000000000002e-05 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:240 3.2 e38/30 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.858360011, 2024-09-13T11:32:42.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028057044, 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166000+0000", 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179000+0000", 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013000523999999999 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179014+0000", 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4143e-05 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179023+0000", 2024-09-13T11:32:42.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5969999999999992e-06 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179083+0000", 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9911000000000003e-05 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179110+0000", 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7251999999999999e-05 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014884616999999999 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.194055+0000", 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:246 3.2 e38/30 3:68241bcb:::smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'6)", 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.836848381999999, 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013298119000000001, 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187511+0000", 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000003e-07 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197044+0000", 2024-09-13T11:32:42.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095328879999999998 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197061+0000", 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7413000000000001e-05 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197072+0000", 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1078e-05 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197146+0000", 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4449999999999994e-05 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197176+0000", 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9584e-05 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.200753+0000", 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035767020000000002 2024-09-13T11:32:42.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.200808+0000", 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5120000000000001e-05 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:244 3.2 e38/30 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'6)", 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.827247471, 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023733269000000001, 2024-09-13T11:32:42.095 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204547+0000", 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074358269999999999 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204567+0000", 2024-09-13T11:32:42.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0196000000000001e-05 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204579+0000", 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2071e-05 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204679+0000", 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9702000000000002e-05 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204727+0000", 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7929000000000001e-05 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220809+0000", 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016082012999999999 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220844+0000", 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4959999999999997e-05 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.099 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:248 3.2 e38/30 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'11, mlcod=38'8)", 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.819746263000001, 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016257252999999999, 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:42.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204613+0000", 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5300000000000003e-07 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210186+0000", 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005573789 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210202+0000", 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5424999999999999e-05 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210215+0000", 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3393e-05 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210285+0000", 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9295000000000007e-05 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210305+0000", 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0557999999999999e-05 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220848+0000", 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010542463 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220869+0000", 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1577e-05 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:247 3.2 e38/30 3:4f19c0b8:::smithi20438004-48 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.814115118, 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016148066999999999, 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.104 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210244+0000", 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4099999999999997e-07 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217679+0000", 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074353470000000001 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217694+0000", 2024-09-13T11:32:42.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.455e-05 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217702+0000", 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217760+0000", 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7788000000000003e-05 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217786+0000", 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5803999999999999e-05 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226348+0000", 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085624180000000005 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226391+0000", 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2660999999999997e-05 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.108 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1351 4.d 4:b7c9e12a:::smithi20438004-12:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:35.791077+0000", 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.2332814169999997, 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022759601000000001, 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:42.109 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1351 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791077+0000", 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791077+0000", 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791079+0000", 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5280000000000002e-06 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791090+0000", 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0507e-05 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791112+0000", 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2364000000000001e-05 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791125+0000", 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2434e-05 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791200+0000", 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5482999999999995e-05 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791297+0000", 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7058999999999993e-05 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.813836+0000", 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022539225999999999 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.841766+0000", 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.1825916620000001, 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019390853999999999, 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.113 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.841766+0000", 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.841766+0000", 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.841767+0000", 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1400000000000001e-07 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855528+0000", 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013761379000000001 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855543+0000", 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5065999999999999e-05 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855552+0000", 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8459999999999997e-06 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855603+0000", 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1294000000000001e-05 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855621+0000", 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7482999999999998e-05 2024-09-13T11:32:42.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.861105+0000", 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054846870000000002 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.861157+0000", 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1885000000000002e-05 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:1651 3.2 e45/30 3:720dc439:::2c9d97cd8bc343ec27e96488c4591170a3e62f503f5d642c2598a4d2866780e1:head v 45'112, mlcod=45'110)", 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:41.057748+0000", 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.96661012000000002, 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013229449000000001, 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.057748+0000", 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.057748+0000", 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.057751+0000", 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8729999999999999e-06 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062779+0000", 2024-09-13T11:32:42.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050277100000000003 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062814+0000", 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5336000000000002e-05 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062845+0000", 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0969000000000001e-05 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062906+0000", 2024-09-13T11:32:42.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1548000000000001e-05 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062925+0000", 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8669e-05 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.070940+0000", 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0080148179999999999 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.070977+0000", 2024-09-13T11:32:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7526e-05 2024-09-13T11:32:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.122 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.122 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:892: finishing write tid 2 to smithi20438004-24 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:892: finishing write tid 3 to smithi20438004-24 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:881: done (15 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:882: done (14 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:883: done (13 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:884: done (12 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:885: done (11 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:886: done (10 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:887: done (9 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:888: done (8 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:889: done (7 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:890: done (6 left) 2024-09-13T11:32:42.124 INFO:tasks.rados.rados.0.smithi204.stdout:891: done (5 left) 2024-09-13T11:32:42.125 INFO:tasks.rados.rados.0.smithi204.stdout:893: done (4 left) 2024-09-13T11:32:42.125 INFO:tasks.rados.rados.0.smithi204.stdout:897: read oid 30 snap -1 2024-09-13T11:32:42.125 INFO:tasks.rados.rados.0.smithi204.stdout:897: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:32:42.125 INFO:tasks.rados.rados.0.smithi204.stdout:898: set_chunk oid 23 target oid 2024-09-13T11:32:42.126 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:32:42.127 INFO:tasks.rados.rados.0.smithi204.stdout:898: set_chunk oid 23 offset: 1244672 length: 2563584 target oid 9cdff82839ec02cbefa36e556cd7a99107b850595852a5fde8d67fefce8c71bc offset: 0 snap -1 2024-09-13T11:32:42.127 INFO:tasks.rados.rados.0.smithi204.stdout:899: tier_flush oid 40 2024-09-13T11:32:42.127 INFO:tasks.rados.rados.0.smithi204.stdout:899: tier_flush oid 40 snap -1 2024-09-13T11:32:42.127 INFO:tasks.rados.rados.0.smithi204.stdout:892: finishing write tid 4 to smithi20438004-24 2024-09-13T11:32:42.127 INFO:tasks.rados.rados.0.smithi204.stdout:892: oid 24 updating version 0 to 61 2024-09-13T11:32:42.127 INFO:tasks.rados.rados.0.smithi204.stdout:892: oid 24 version 61 is already newer than 59 2024-09-13T11:32:42.127 INFO:tasks.rados.rados.0.smithi204.stdout:892: oid 24 version 61 is already newer than 60 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 61 (ObjNum 135 snap 0 seq_num 135) dirty exists 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:892: left oid 24 (ObjNum 135 snap 0 seq_num 135) 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:894: finishing write tid 1 to smithi20438004-26 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:894: finishing write tid 2 to smithi20438004-26 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:894: finishing write tid 3 to smithi20438004-26 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:892: done (6 left) 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:895: done (5 left) 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:900: read oid 8 snap -1 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:900: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:901: delete oid 48 current snap is 0 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:894: finishing write tid 4 to smithi20438004-26 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:894: oid 26 updating version 0 to 19 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:894: oid 26 updating version 19 to 20 2024-09-13T11:32:42.128 INFO:tasks.rados.rados.0.smithi204.stdout:894: oid 26 version 20 is already newer than 18 2024-09-13T11:32:42.129 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 20 (ObjNum 136 snap 0 seq_num 136) dirty exists 2024-09-13T11:32:42.129 INFO:tasks.rados.rados.0.smithi204.stdout:894: left oid 26 (ObjNum 136 snap 0 seq_num 136) 2024-09-13T11:32:42.129 INFO:tasks.rados.rados.0.smithi204.stdout:896: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-13T11:32:42.129 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:42.122+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:42.164 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:42.215 INFO:tasks.rados.rados.0.smithi204.stdout:897: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:32:42.215 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:42.223 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:42.222+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:42.263 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2003515 2024-09-13T11:32:42.267 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:42.267 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:42.268 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:42.268 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:42.276 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:32:42.298 INFO:tasks.rados.rados.0.smithi204.stdout:898: finishing set_chunk to oid 23 2024-09-13T11:32:42.298 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 45 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-09-13T11:32:42.298 INFO:tasks.rados.rados.0.smithi204.stdout:900: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-13T11:32:42.325 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:42.324+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:42.342 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:42.367 INFO:tasks.rados.rados.0.smithi204.stdout:894: done (6 left) 2024-09-13T11:32:42.367 INFO:tasks.rados.rados.0.smithi204.stdout:896: done (5 left) 2024-09-13T11:32:42.367 INFO:tasks.rados.rados.0.smithi204.stdout:897: done (4 left) 2024-09-13T11:32:42.367 INFO:tasks.rados.rados.0.smithi204.stdout:898: done (3 left) 2024-09-13T11:32:42.367 INFO:tasks.rados.rados.0.smithi204.stdout:900: done (2 left) 2024-09-13T11:32:42.367 INFO:tasks.rados.rados.0.smithi204.stdout:901: done (1 left) 2024-09-13T11:32:42.367 INFO:tasks.rados.rados.0.smithi204.stdout:902: tier_evict oid 6 2024-09-13T11:32:42.368 INFO:tasks.rados.rados.0.smithi204.stdout:902: tier_evict oid 6 snap -1 2024-09-13T11:32:42.368 INFO:tasks.rados.rados.0.smithi204.stdout:903: set_chunk oid 36 target oid 2024-09-13T11:32:42.368 INFO:tasks.rados.rados.0.smithi204.stdout:903: set_chunk oid 36 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:42.368 INFO:tasks.rados.rados.0.smithi204.stdout:904: delete oid 42 current snap is 0 2024-09-13T11:32:42.368 INFO:tasks.rados.rados.0.smithi204.stdout:902: got (22) Invalid argument 2024-09-13T11:32:42.368 INFO:tasks.rados.rados.0.smithi204.stdout:903: finishing set_chunk to oid 36 2024-09-13T11:32:42.368 INFO:tasks.rados.rados.0.smithi204.stdout:903: got expected ENOENT (src dne) 2024-09-13T11:32:42.369 INFO:tasks.rados.rados.0.smithi204.stdout:902: done (3 left) 2024-09-13T11:32:42.369 INFO:tasks.rados.rados.0.smithi204.stdout:903: done (2 left) 2024-09-13T11:32:42.369 INFO:tasks.rados.rados.0.smithi204.stdout:904: done (1 left) 2024-09-13T11:32:42.369 INFO:tasks.rados.rados.0.smithi204.stdout:905: read oid 38 snap -1 2024-09-13T11:32:42.369 INFO:tasks.rados.rados.0.smithi204.stdout:905: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-13T11:32:42.369 INFO:tasks.rados.rados.0.smithi204.stdout:906: set_chunk oid 9 target oid 2024-09-13T11:32:42.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:42.400 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:42.400 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:42.400 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:42.400 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:42.401 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:42.408 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:32:42.447 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:42.446+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:42.512 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:42.512 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:42.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:42.512 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:42.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.512 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:42.512 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.413026502999999, 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:32:42.516 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.400592358000001, 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.390638764, 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.522 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:224 3.3 e38/30 3:e7e3acce:::smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=38'1)", 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.365999518000001, 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010654663, 2024-09-13T11:32:42.527 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:42.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144201+0000", 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1969999999999999e-06 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150234+0000", 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060323859999999998 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150248+0000", 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4584e-05 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150256+0000", 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1270000000000003e-06 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150306+0000", 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9633999999999999e-05 2024-09-13T11:32:42.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150327+0000", 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0579999999999999e-05 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154794+0000", 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004467108 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154854+0000", 2024-09-13T11:32:42.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0047000000000003e-05 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.359920957, 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.532 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:42.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:32:42.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:32:42.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:32:42.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.341097638000001, 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.538 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:32:42.539 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:32:42.540 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.541 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.542 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.327578289, 2024-09-13T11:32:42.543 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.544 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:32:42.545 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.546 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.547 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.548 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.317937956, 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.549 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.550 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.551 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.552 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:32:42.553 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.307632456, 2024-09-13T11:32:42.554 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.555 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:32:42.556 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:32:42.558 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:42.559 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.297246513999999, 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.560 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:32:42.561 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.562 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:32:42.563 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:32:42.564 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.564 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.564 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.564 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:32:42.564 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:32:42.564 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.564 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.3.22:1 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.288479812, 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0048600099999999997, 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.565 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.566 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221719+0000", 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221724+0000", 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7090000000000001e-06 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221732+0000", 2024-09-13T11:32:42.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1599999999999998e-06 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221737+0000", 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3939999999999998e-06 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221777+0000", 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0666e-05 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221789+0000", 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1986e-05 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226541+0000", 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047513249999999998 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.569 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226579+0000", 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8158999999999998e-05 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.12.22:18 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'22, mlcod=38'19)", 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.253030+0000", 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.25716819, 2024-09-13T11:32:42.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0051860980000000001, 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253030+0000", 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253030+0000", 2024-09-13T11:32:42.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253031+0000", 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3300000000000002e-07 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253036+0000", 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2480000000000004e-06 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253045+0000", 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9719999999999998e-06 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253052+0000", 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.409e-06 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253113+0000", 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1438000000000004e-05 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253133+0000", 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9823e-05 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.258182+0000", 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050486339999999998 2024-09-13T11:32:42.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.258217+0000", 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4941000000000003e-05 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.9.22:9 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'24, mlcod=38'21)", 2024-09-13T11:32:42.575 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.242832823000001, 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0049262630000000002, 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267366+0000", 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3099999999999999e-07 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267370+0000", 2024-09-13T11:32:42.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1740000000000002e-06 2024-09-13T11:32:42.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267376+0000", 2024-09-13T11:32:42.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1750000000000002e-06 2024-09-13T11:32:42.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267381+0000", 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9930000000000003e-06 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267656+0000", 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00027412500000000001 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267676+0000", 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0488000000000001e-05 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272254+0000", 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0045782499999999999 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272292+0000", 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7727e-05 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.12.22:19 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'25, mlcod=38'21)", 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.267885+0000", 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.242313857999999, 2024-09-13T11:32:42.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0044361610000000001, 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267885+0000", 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267885+0000", 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267886+0000", 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267893+0000", 2024-09-13T11:32:42.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8269999999999998e-06 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267906+0000", 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3152e-05 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267912+0000", 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9880000000000001e-06 2024-09-13T11:32:42.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.267993+0000", 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0130999999999996e-05 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.268020+0000", 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6984000000000001e-05 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272295+0000", 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004275203 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.272321+0000", 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6095999999999999e-05 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:772 3.3 e38/30 3:dd3ad6bf:::4ec2dd001b7153a0bb00c76a8320dec150c3eb87238ccdd9135be5e948aee2d9:head v 38'30, mlcod=38'28)", 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.359518+0000", 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.150680959000001, 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068126380000000002, 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.359518+0000", 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.359518+0000", 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.588 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.359519+0000", 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5659999999999999e-06 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362100+0000", 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025806969999999999 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362119+0000", 2024-09-13T11:32:42.589 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9320000000000001e-05 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362131+0000", 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1438e-05 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362189+0000", 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8730999999999997e-05 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.362207+0000", 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7655999999999998e-05 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.366279+0000", 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0040718969999999997 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.366330+0000", 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1332999999999998e-05 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.3.22:17 3.3 e43/41 3:ef23b735:::2bd8d6c960c353b6c5e5df2c9cfadedb8591eb771e008699ee9f872652738b62:head v 43'72, mlcod=43'70)", 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:35.807412+0000", 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.7027861160000004, 2024-09-13T11:32:42.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004859337, 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.807412+0000", 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.807412+0000", 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.807414+0000", 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3540000000000001e-06 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.808312+0000", 2024-09-13T11:32:42.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089838600000000004 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.808330+0000", 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7577000000000001e-05 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.808339+0000", 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0420000000000008e-06 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.808415+0000", 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6062999999999993e-05 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.808436+0000", 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1594e-05 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.812231+0000", 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003794665 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.812272+0000", 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0655999999999998e-05 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:1423 4.0 e44/31 4:04aeeab9:::smithi20438004-5:head v 44'32, mlcod=43'30)", 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.776364+0000", 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.7338350709999997, 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047840180000000001, 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.776364+0000", 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.776364+0000", 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.776365+0000", 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.809e-06 2024-09-13T11:32:42.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.777334+0000", 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096894799999999997 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.777356+0000", 2024-09-13T11:32:42.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1812999999999999e-05 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.777368+0000", 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1545999999999999e-05 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.777436+0000", 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8707000000000005e-05 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.777463+0000", 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6434000000000001e-05 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.781118+0000", 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036551460000000002 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.781148+0000", 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9615e-05 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:1480 3.3 e44/41 3:fc8d920c:::574e186947520a2389cbaeea8ba1176542b7ebf6bc2d1924ea8fea4e9a7674ef:head v 44'83, mlcod=44'81)", 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.100870+0000", 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.4093285309999999, 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087291580000000007, 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.603 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.100870+0000", 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.100870+0000", 2024-09-13T11:32:42.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.100872+0000", 2024-09-13T11:32:42.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5370000000000001e-06 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104077+0000", 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032057600000000002 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104088+0000", 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0220999999999999e-05 2024-09-13T11:32:42.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104097+0000", 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3000000000000007e-06 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104183+0000", 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5964999999999994e-05 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104203+0000", 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0202000000000001e-05 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.109556+0000", 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053534350000000001 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.109599+0000", 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.612 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:1585 4.0 e45/31 4:04aeeab9:::smithi20438004-5:head v 45'38, mlcod=44'36)", 2024-09-13T11:32:42.612 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:40.294217+0000", 2024-09-13T11:32:42.612 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 2.2159816870000002, 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053848990000000003, 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.294217+0000", 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.294217+0000", 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.294217+0000", 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8400000000000001e-07 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.295511+0000", 2024-09-13T11:32:42.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001294164 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.295522+0000", 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0689999999999999e-05 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.295528+0000", 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.105e-06 2024-09-13T11:32:42.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.295579+0000", 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0738000000000001e-05 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.295604+0000", 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5355999999999999e-05 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.299564+0000", 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003960285 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:40.299602+0000", 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7376999999999998e-05 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:1677 3.3 e46/41 3:e8b97dbc:::9cdff82839ec02cbefa36e556cd7a99107b850595852a5fde8d67fefce8c71bc:head v 46'94, mlcod=45'92)", 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:42.052449+0000", 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 0.45774972800000002, 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056205389999999999, 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.052449+0000", 2024-09-13T11:32:42.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.052449+0000", 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.052451+0000", 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3999999999999999e-06 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.055318+0000", 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0028671790000000001 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.055337+0000", 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8658e-05 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:42.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.055349+0000", 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1953999999999999e-05 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.055439+0000", 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0377999999999994e-05 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.055461+0000", 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1622000000000001e-05 2024-09-13T11:32:42.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.058032+0000", 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025711660000000002 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:42.058069+0000", 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7181999999999997e-05 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.622 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.623 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:42.623 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:42.623 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:42.624 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:32:42.624 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.5. 2024-09-13T11:32:42.625 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:42.625 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:42.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:32:42.627 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:42.623+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:42.628 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-13T11:32:42.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 9 2024-09-13T11:32:42.629 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:42.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:42.629 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:42.629 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:42.630 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:42.630 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:42.630 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:42.631 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:42.631 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:42.631 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:42.631 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:42.645 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:42.646 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:42.662 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:42.663 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:42.680 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:42.680 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:42.698 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:42.698 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:42.714 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:42.715 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:42.725 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:42.724+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:42.731 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:42.731 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:42.749 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:42.749 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:42.766 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:42.767 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:42.771 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:42.771 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:42.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:32:42.783 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:42.783 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:42.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:42.864 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:42.863+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:42.924 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:42.924 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:42.927 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:42.926+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:42.932 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:32:42.959 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:32:42.957+0000 7f8352d34640 -1 mon.a@0(leader).osd e47 definitely_dead 0 2024-09-13T11:32:43.028 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:43.027+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:43.078 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:43.079 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:43.086 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:32:43.129 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:43.128+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:43.189 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:43.190 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:43.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:32:43.230 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:43.229+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:43.302 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:43.302 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:43.311 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:32:43.332 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:43.331+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:43.413 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:43.413 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:43.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:32:43.432 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:43.431+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:43.533 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:43.532+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:43.534 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:43.534 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:43.540 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.9. 2024-09-13T11:32:43.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:32:43.557 INFO:tasks.thrashosds.thrasher:Removing osd 13, in_osds are: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 15] 2024-09-13T11:32:43.557 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 13 2024-09-13T11:32:43.642 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:43.641+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:43.716 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:43.717 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:43.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:32:43.736 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:43.735+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:43.837 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:43.836+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:43.874 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:43.875 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:43.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:32:43.920 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:32:43.917+0000 7f8352d34640 -1 mon.a@0(leader).osd e48 definitely_dead 0 2024-09-13T11:32:43.938 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:43.937+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:44.027 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:44.027 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:44.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:32:44.079 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:44.078+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:44.139 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:44.139+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:44.142 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:44.142 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:44.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:32:44.241 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:44.240+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:44.299 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:44.299 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:44.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:32:44.341 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:44.340+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:44.442 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:44.441+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:44.456 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:44.456 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:44.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:32:44.539 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.13. 2024-09-13T11:32:44.543 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:44.542+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:44.623 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:44.623 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:44.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:32:44.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:32:44.680 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:44.678+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:44.745 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:44.743+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:44.788 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:44.789 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:44.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:44.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:32:44.846 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:44.844+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:44.920 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:44.921 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:44.938 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:44.939 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:44.939 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:44.946 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:44.944+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:44.949 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:44.949 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:44.957 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:44.957 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:44.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:32:44.973 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:44.973 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:44.991 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:44.991 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:45.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:45.026 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:45.026 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:45.043 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:45.043 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:45.047 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:45.046+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:45.060 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:45.061 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:45.066 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:45.066 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:45.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:32:45.078 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:45.080 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:45.095 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:45.114 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:45.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:45.131 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:45.132 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:45.148 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:45.146+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:45.149 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:45.149 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:45.182 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:45.182 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:45.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:204 3.0 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.115509823, 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025518434, 2024-09-13T11:32:45.202 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 204 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084160+0000", 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.463e-06 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084171+0000", 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1206000000000001e-05 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084199+0000", 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7796e-05 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084215+0000", 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5820000000000001e-05 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084389+0000", 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174134 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084589+0000", 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020036899999999999 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.091157+0000", 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065682249999999996 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.096484+0000", 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053268400000000002 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.102639+0000", 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061551430000000001 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109591+0000", 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006951774 2024-09-13T11:32:45.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109627+0000", 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6198999999999997e-05 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109676+0000", 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8464999999999998e-05 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:205 3.0 3:03052d75:::smithi20438004-6 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.115421539, 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.046448324999999999, 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 205 2024-09-13T11:32:45.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8999999999999995e-07 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084251+0000", 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9899999999999997e-06 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084262+0000", 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.023e-05 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084270+0000", 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0669999999999992e-06 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084746+0000", 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047623199999999999 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084842+0000", 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6065999999999998e-05 2024-09-13T11:32:45.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.094307+0000", 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094653670000000006 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.098690+0000", 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043829810000000002 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.107503+0000", 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088122559999999992 2024-09-13T11:32:45.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130597+0000", 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023094078000000001 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130642+0000", 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5463999999999997e-05 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130694+0000", 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1703999999999998e-05 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:213 3.0 3:22c1b591:::smithi20438004-14 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.115229495000001, 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.055117410999999998, 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 213 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5200000000000002e-07 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084441+0000", 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:32:45.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084447+0000", 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5430000000000001e-06 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084451+0000", 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5829999999999998e-06 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084997+0000", 2024-09-13T11:32:45.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054659399999999996 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085141+0000", 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014359400000000001 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.097495+0000", 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012354146999999999 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.099703+0000", 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022078079999999999 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.112967+0000", 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013264582 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139474+0000", 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026506980999999999 2024-09-13T11:32:45.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139513+0000", 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8322999999999999e-05 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139555+0000", 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2475999999999998e-05 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:217 3.0 3:33b1e45a:::smithi20438004-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.091207014999998, 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03827937, 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 217 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.222 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108461+0000", 2024-09-13T11:32:45.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.392e-06 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108467+0000", 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8429999999999998e-06 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108478+0000", 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0243e-05 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108483+0000", 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8000000000000004e-06 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108545+0000", 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1909000000000006e-05 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108674+0000", 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012890699999999999 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.120164+0000", 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011489868 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.122425+0000", 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002260649 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.125843+0000", 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034186220000000001 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146587+0000", 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020743276000000001 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146664+0000", 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7330999999999995e-05 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146739+0000", 2024-09-13T11:32:45.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5530000000000004e-05 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:222 3.0 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.080329999, 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033143700999999998, 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.228 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 222 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119339+0000", 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.452e-06 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119345+0000", 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8499999999999996e-06 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119358+0000", 2024-09-13T11:32:45.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119364+0000", 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3219999999999998e-06 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119411+0000", 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6186999999999997e-05 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119552+0000", 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000141204 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.128873+0000", 2024-09-13T11:32:45.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093210610000000003 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.131258+0000", 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023855529999999999 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.136698+0000", 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005439854 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152381+0000", 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015683002000000001 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152419+0000", 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7561000000000002e-05 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152481+0000", 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2038000000000005e-05 2024-09-13T11:32:45.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:225 3.0 3:10cf6cc6:::smithi20438004-26 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.066103475999999, 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026005851, 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.235 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 225 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133565+0000", 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.052e-06 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133571+0000", 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9349999999999999e-06 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133580+0000", 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8530000000000004e-06 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133586+0000", 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9610000000000002e-06 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133632+0000", 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5427999999999999e-05 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133735+0000", 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102829 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147872+0000", 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014137244 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152657+0000", 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047851830000000001 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159315+0000", 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066580700000000003 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159445+0000", 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130104 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159513+0000", 2024-09-13T11:32:45.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7897000000000004e-05 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159569+0000", 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6295000000000002e-05 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:227 3.0 3:162d2d40:::smithi20438004-28 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.059902164, 2024-09-13T11:32:45.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028012617, 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139766+0000", 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139770+0000", 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0149999999999997e-06 2024-09-13T11:32:45.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139779+0000", 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4460000000000007e-06 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139784+0000", 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8779999999999997e-06 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139822+0000", 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7750999999999998e-05 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139934+0000", 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112378 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.150743+0000", 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010808768999999999 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.154264+0000", 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035207160000000001 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.164397+0000", 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010133675999999999 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167661+0000", 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032640759999999999 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167728+0000", 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7018000000000001e-05 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167778+0000", 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9197999999999999e-05 2024-09-13T11:32:45.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:232 3.0 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.052135921000001, 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032659146999999999, 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.248 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 232 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:45.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147533+0000", 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147539+0000", 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8850000000000004e-06 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147555+0000", 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6065e-05 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147568+0000", 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3236e-05 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152097+0000", 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045292199999999996 2024-09-13T11:32:45.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152211+0000", 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001141 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.162843+0000", 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010631982999999999 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167140+0000", 2024-09-13T11:32:45.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042969829999999999 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.174951+0000", 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078107159999999997 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180068+0000", 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051167369999999997 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180131+0000", 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3140999999999997e-05 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180190+0000", 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9438999999999998e-05 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:234 3.0 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:45.254 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.051354594999999, 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036893333, 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 234 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148314+0000", 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.871e-06 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148323+0000", 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5010000000000004e-06 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148334+0000", 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1280999999999999e-05 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148342+0000", 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152513+0000", 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041709370000000004 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152594+0000", 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0315999999999995e-05 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165507+0000", 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012913051 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.168150+0000", 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026431200000000001 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179750+0000", 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01160015 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185129+0000", 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053793729999999998 2024-09-13T11:32:45.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185171+0000", 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2073000000000002e-05 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4456999999999999e-05 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:233 3.2 e38/30 3:53d0adee:::smithi20438004-34 low_tier:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.046138225, 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018647573000000001, 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153530+0000", 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4920000000000001e-06 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169595+0000", 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016064407999999999 2024-09-13T11:32:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169603+0000", 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7250000000000003e-06 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169609+0000", 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3900000000000001e-06 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169669+0000", 2024-09-13T11:32:45.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0052e-05 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169684+0000", 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4778e-05 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.172138+0000", 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024544110000000001 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.172176+0000", 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8316999999999999e-05 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:239 3.0 3:3c12287e:::smithi20438004-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.039587461, 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037773656000000003, 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 239 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:45.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160086+0000", 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6810000000000003e-06 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160097+0000", 2024-09-13T11:32:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.101e-05 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160103+0000", 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6319999999999996e-06 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160148+0000", 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.511e-05 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160285+0000", 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000137508 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.175532+0000", 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015246902999999999 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180030+0000", 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044977300000000001 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185903+0000", 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058733659999999997 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197739+0000", 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011835094000000001 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197801+0000", 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2226999999999999e-05 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197853+0000", 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2537999999999997e-05 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:241 3.0 3:2c9c5010:::smithi20438004-42 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.033817955, 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047296003000000003, 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 241 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165851+0000", 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5269999999999999e-06 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165867+0000", 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6827e-05 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165880+0000", 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165890+0000", 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0101999999999999e-05 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167178+0000", 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00128834 2024-09-13T11:32:45.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167324+0000", 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014554300000000001 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.178258+0000", 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010933951000000001 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180829+0000", 2024-09-13T11:32:45.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025710849999999999 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.186910+0000", 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060810639999999997 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213073+0000", 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.02616305 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213112+0000", 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9292000000000003e-05 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213145+0000", 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3126000000000001e-05 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:240 3.2 e38/30 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:45.277 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.030032197000001, 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014316538, 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169635+0000", 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3599999999999999e-07 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181076+0000", 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011440584 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181086+0000", 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0138e-05 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181092+0000", 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8409999999999996e-06 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181142+0000", 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9694999999999998e-05 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.181167+0000", 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5499000000000001e-05 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.183914+0000", 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027467220000000001 2024-09-13T11:32:45.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.183951+0000", 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7623000000000002e-05 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:243 3.0 3:03808b00:::smithi20438004-44 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.022707314000002, 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042573002999999998, 2024-09-13T11:32:45.282 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 243 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176961+0000", 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6610000000000001e-06 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176970+0000", 2024-09-13T11:32:45.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2779999999999993e-06 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176982+0000", 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2711000000000001e-05 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176990+0000", 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000003e-06 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179794+0000", 2024-09-13T11:32:45.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028036379999999998 2024-09-13T11:32:45.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179911+0000", 2024-09-13T11:32:45.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117095 2024-09-13T11:32:45.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.195122+0000", 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015211107 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197868+0000", 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027461540000000002 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.204148+0000", 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006280435 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219448+0000", 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015299787 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3766000000000001e-05 2024-09-13T11:32:45.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219533+0000", 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0880999999999997e-05 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:251 4.6 4:6d3e8486:::smithi20438004-2:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.001821842999998, 2024-09-13T11:32:45.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021332106999999999, 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 251 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197847+0000", 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197853+0000", 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0530000000000003e-06 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197864+0000", 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0961999999999999e-05 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197874+0000", 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0377000000000001e-05 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197920+0000", 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5178e-05 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197996+0000", 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6373999999999998e-05 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218088+0000", 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020092445 2024-09-13T11:32:45.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218692+0000", 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060317200000000002 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219008+0000", 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000316328 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219097+0000", 2024-09-13T11:32:45.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8993000000000007e-05 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219140+0000", 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3167000000000002e-05 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219177+0000", 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7301000000000001e-05 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:253 4.6 4:600f3d3f:::smithi20438004-4:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.000115339000001, 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022588031000000001, 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199553+0000", 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9299999999999996e-07 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199557+0000", 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6110000000000004e-06 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199568+0000", 2024-09-13T11:32:45.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0582e-05 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199574+0000", 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3500000000000002e-06 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199623+0000", 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8736999999999998e-05 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199673+0000", 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0232999999999998e-05 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219425+0000", 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019751614000000001 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219918+0000", 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00049309600000000005 2024-09-13T11:32:45.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221826+0000", 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019085479999999999 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222084+0000", 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025730000000000002 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222114+0000", 2024-09-13T11:32:45.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0048000000000001e-05 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222140+0000", 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6018999999999999e-05 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.10.22:1 3.0 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.996391500000001, 2024-09-13T11:32:45.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016452626000000001, 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203277+0000", 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-06 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203282+0000", 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.882e-06 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203290+0000", 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203300+0000", 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0162e-05 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203350+0000", 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9487999999999998e-05 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203422+0000", 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2537e-05 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203577+0000", 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015495999999999999 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.204164+0000", 2024-09-13T11:32:45.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058732999999999995 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.205037+0000", 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087292200000000004 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219676+0000", 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014638463000000001 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:45.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219704+0000", 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8396999999999999e-05 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219728+0000", 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3944e-05 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.15.23:64 3.0 3.3342ca04 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:32:45.309 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:42.060517+0000", 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.139150168, 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.47916420900000001, 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060517+0000", 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060517+0000", 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060519+0000", 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.939e-06 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060526+0000", 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5320000000000001e-06 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060542+0000", 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5852e-05 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060551+0000", 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1870000000000003e-06 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060628+0000", 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6428999999999996e-05 2024-09-13T11:32:45.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060635+0000", 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539649+0000", 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.47901452 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539681+0000", 2024-09-13T11:32:45.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1897000000000003e-05 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.15.23:65 3.0 3.a092811c (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:42.060578+0000", 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1390890599999999, 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.479226188, 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:32:45.314 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060578+0000", 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060578+0000", 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060579+0000", 2024-09-13T11:32:45.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0399999999999996e-07 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060582+0000", 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7680000000000002e-06 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060592+0000", 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3980000000000005e-06 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060599+0000", 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2339999999999997e-06 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060660+0000", 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0534e-05 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060666+0000", 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7859999999999999e-06 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539795+0000", 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.47912892499999998 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539804+0000", 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9390000000000003e-06 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1690 4.3 4.db1fe8e3 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:42.368473+0000", 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.8311945989999998, 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17124435099999999, 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1690 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:45.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368473+0000", 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368473+0000", 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368474+0000", 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.595e-06 2024-09-13T11:32:45.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368479+0000", 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2830000000000001e-06 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368492+0000", 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.295e-05 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368509+0000", 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6541e-05 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368565+0000", 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5868000000000001e-05 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368573+0000", 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6929999999999996e-06 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:45.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539669+0000", 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17109563999999999 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539717+0000", 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7781000000000002e-05 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:45.323 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:45.325 INFO:tasks.rados.rados.0.smithi204.stdout:906: set_chunk oid 9 offset: 2345984 length: 79360 target oid c2f3cf71c5c10ef5a6390c59974ee92cd11d862a6a7bf1b93dd42e0094795f21 offset: 0 snap -1 2024-09-13T11:32:45.325 INFO:tasks.rados.rados.0.smithi204.stdout:907: read oid 41 snap -1 2024-09-13T11:32:45.325 INFO:tasks.rados.rados.0.smithi204.stdout:907: expect (ObjNum 127 snap 0 seq_num 127) 2024-09-13T11:32:45.326 INFO:tasks.rados.rados.0.smithi204.stdout:908: tier_promote oid 50 2024-09-13T11:32:45.326 INFO:tasks.rados.rados.0.smithi204.stdout:905: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-13T11:32:45.326 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:45.326 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:45.323+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:45.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:32:45.327 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:32:45.384 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1937030 2024-09-13T11:32:45.387 INFO:tasks.rados.rados.0.smithi204.stdout:906: finishing set_chunk to oid 9 2024-09-13T11:32:45.387 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 32 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-13T11:32:45.387 INFO:tasks.rados.rados.0.smithi204.stdout:905: done (4 left) 2024-09-13T11:32:45.387 INFO:tasks.rados.rados.0.smithi204.stdout:906: done (3 left) 2024-09-13T11:32:45.387 INFO:tasks.rados.rados.0.smithi204.stdout:909: set_chunk oid 42 target oid 2024-09-13T11:32:45.387 INFO:tasks.rados.rados.0.smithi204.stdout:909: set_chunk oid 42 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:45.388 INFO:tasks.rados.rados.0.smithi204.stdout:910: set_chunk oid 46 target oid 2024-09-13T11:32:45.395 INFO:tasks.rados.rados.0.smithi204.stdout:910: set_chunk oid 46 offset: 1762304 length: 809472 target oid 3642d573c78be90b2c54aedaec3513d4b5c0eaf4cb10598ce407a4765ea1bd5c offset: 0 snap -1 2024-09-13T11:32:45.395 INFO:tasks.rados.rados.0.smithi204.stdout:911: write oid 17 current snap is 0 2024-09-13T11:32:45.395 INFO:tasks.rados.rados.0.smithi204.stdout:909: finishing set_chunk to oid 42 2024-09-13T11:32:45.395 INFO:tasks.rados.rados.0.smithi204.stdout:909: got expected ENOENT (src dne) 2024-09-13T11:32:45.395 INFO:tasks.rados.rados.0.smithi204.stdout:911: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-13T11:32:45.401 INFO:tasks.rados.rados.0.smithi204.stdout:911: writing smithi20438004-17 from 736173 to 1290138 tid 1 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:911: writing smithi20438004-17 from 1867350 to 2336173 tid 2 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:909: done (5 left) 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:912: write (excl) oid 29 current snap is 0 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:910: finishing set_chunk to oid 46 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 1762304 length: 809472 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:911: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:45.408 INFO:tasks.rados.rados.0.smithi204.stdout:912: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-13T11:32:45.415 INFO:tasks.rados.rados.0.smithi204.stdout:912: writing smithi20438004-29 from 752980 to 1382194 tid 1 2024-09-13T11:32:45.421 INFO:tasks.rados.rados.0.smithi204.stdout:912: writing smithi20438004-29 from 1809479 to 2352980 tid 2 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:910: done (5 left) 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:913: read oid 2 snap -1 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:913: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:914: read oid 23 snap -1 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:914: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:915: write (excl) oid 39 current snap is 0 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:911: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:911: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:911: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:45.422 INFO:tasks.rados.rados.0.smithi204.stdout:911: oid 17 updating version 0 to 55 2024-09-13T11:32:45.423 INFO:tasks.rados.rados.0.smithi204.stdout:911: oid 17 updating version 55 to 56 2024-09-13T11:32:45.423 INFO:tasks.rados.rados.0.smithi204.stdout:911: oid 17 version 56 is already newer than 54 2024-09-13T11:32:45.423 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 56 (ObjNum 139 snap 0 seq_num 139) dirty exists 2024-09-13T11:32:45.423 INFO:tasks.rados.rados.0.smithi204.stdout:911: left oid 17 (ObjNum 139 snap 0 seq_num 139) 2024-09-13T11:32:45.423 INFO:tasks.rados.rados.0.smithi204.stdout:912: finishing write tid 1 to smithi20438004-29 2024-09-13T11:32:45.423 INFO:tasks.rados.rados.0.smithi204.stdout:915: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-13T11:32:45.425 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:45.424+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:45.429 INFO:tasks.rados.rados.0.smithi204.stdout:915: writing smithi20438004-39 from 769787 to 1474250 tid 1 2024-09-13T11:32:45.434 INFO:tasks.rados.rados.0.smithi204.stdout:915: writing smithi20438004-39 from 2267961 to 2369787 tid 2 2024-09-13T11:32:45.434 INFO:tasks.rados.rados.0.smithi204.stdout:911: done (7 left) 2024-09-13T11:32:45.434 INFO:tasks.rados.rados.0.smithi204.stdout:916: read oid 1 snap -1 2024-09-13T11:32:45.434 INFO:tasks.rados.rados.0.smithi204.stdout:916: expect (ObjNum 103 snap 0 seq_num 103) 2024-09-13T11:32:45.434 INFO:tasks.rados.rados.0.smithi204.stdout:917: read oid 11 snap -1 2024-09-13T11:32:45.434 INFO:tasks.rados.rados.0.smithi204.stdout:917: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-13T11:32:45.435 INFO:tasks.rados.rados.0.smithi204.stdout:918: read oid 44 snap -1 2024-09-13T11:32:45.435 INFO:tasks.rados.rados.0.smithi204.stdout:918: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:45.435 INFO:tasks.rados.rados.0.smithi204.stdout:919: write (excl) oid 5 current snap is 0 2024-09-13T11:32:45.435 INFO:tasks.rados.rados.0.smithi204.stdout:919: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-13T11:32:45.442 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:45.443 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:45.443 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:45.443 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:45.443 INFO:tasks.rados.rados.0.smithi204.stdout:919: writing smithi20438004-5 from 786594 to 1566306 tid 1 2024-09-13T11:32:45.448 INFO:tasks.rados.rados.0.smithi204.stdout:919: writing smithi20438004-5 from 2326443 to 2386594 tid 2 2024-09-13T11:32:45.448 INFO:tasks.rados.rados.0.smithi204.stdout:920: read oid 19 snap -1 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:920: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:921: tier_evict oid 26 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:921: tier_evict oid 26 snap -1 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:912: finishing write tid 2 to smithi20438004-29 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:912: finishing write tid 3 to smithi20438004-29 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:912: finishing write tid 4 to smithi20438004-29 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:912: oid 29 updating version 0 to 63 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:912: oid 29 updating version 63 to 64 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:912: oid 29 version 64 is already newer than 62 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 64 (ObjNum 140 snap 0 seq_num 140) dirty exists 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:912: left oid 29 (ObjNum 140 snap 0 seq_num 140) 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:913: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:45.449 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 127141 2024-09-13T11:32:45.452 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:32:45.487 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 40741 2024-09-13T11:32:45.509 INFO:tasks.rados.rados.0.smithi204.stdout:914: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:45.509 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:45.526 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:45.524+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:45.554 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:45.554 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:45.554 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:45.554 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:45.554 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:45.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:32:45.626 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:45.625+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:45.668 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.575432300999999, 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011006991000000001, 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.669 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:32:45.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091279+0000", 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.091e-06 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098180+0000", 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069015359999999998 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098194+0000", 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.323e-05 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098202+0000", 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2879999999999993e-06 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098267+0000", 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4733999999999994e-05 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098296+0000", 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102225+0000", 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039281940000000003 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102285+0000", 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:32:45.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.518626812000001, 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010863593, 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148085+0000", 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2810000000000001e-06 2024-09-13T11:32:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153254+0000", 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051698430000000004 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153266+0000", 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1973e-05 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153274+0000", 2024-09-13T11:32:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153331+0000", 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6492000000000001e-05 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153396+0000", 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5083e-05 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158906+0000", 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055099570000000002 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158947+0000", 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.513414266000002, 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096623879999999992, 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157238+0000", 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039414139999999999 2024-09-13T11:32:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157253+0000", 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5566e-05 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157260+0000", 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.477e-06 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157315+0000", 2024-09-13T11:32:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4857e-05 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157341+0000", 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162899+0000", 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055583480000000003 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162958+0000", 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9169000000000002e-05 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.503756181, 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011410943999999999, 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162955+0000", 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0669999999999999e-06 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167312+0000", 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004357072 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167333+0000", 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0842000000000001e-05 2024-09-13T11:32:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167341+0000", 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4430000000000003e-06 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167406+0000", 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4245000000000006e-05 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167433+0000", 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7500000000000001e-05 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174280+0000", 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068473969999999999 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174365+0000", 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4377999999999997e-05 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.490928417999999, 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099135219999999993, 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175783+0000", 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179929+0000", 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041461989999999997 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179943+0000", 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4731e-05 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179953+0000", 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:32:45.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180049+0000", 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6909000000000003e-05 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180073+0000", 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4077e-05 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185641+0000", 2024-09-13T11:32:45.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055674330000000001 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185695+0000", 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4273e-05 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:258 4.3 4:c717f8db:::smithi20438004-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.692 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.453161152, 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017162947000000001, 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 258 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213551+0000", 2024-09-13T11:32:45.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0760000000000001e-06 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213566+0000", 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5441999999999999e-05 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213585+0000", 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8623000000000001e-05 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213600+0000", 2024-09-13T11:32:45.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4783999999999999e-05 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213667+0000", 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213747+0000", 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9794000000000002e-05 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228488+0000", 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014740964 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229259+0000", 2024-09-13T11:32:45.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077144599999999996 2024-09-13T11:32:45.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229743+0000", 2024-09-13T11:32:45.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048437000000000003 2024-09-13T11:32:45.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230652+0000", 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090904600000000005 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230688+0000", 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5474e-05 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230712+0000", 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3954000000000001e-05 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:259 4.1 4:8533c921:::smithi20438004-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.44888555, 2024-09-13T11:32:45.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014548484, 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 259 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217826+0000", 2024-09-13T11:32:45.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6360000000000001e-06 2024-09-13T11:32:45.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217833+0000", 2024-09-13T11:32:45.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8700000000000003e-06 2024-09-13T11:32:45.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217844+0000", 2024-09-13T11:32:45.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:32:45.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217851+0000", 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217896+0000", 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4808000000000002e-05 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217954+0000", 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7856000000000003e-05 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229813+0000", 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011859533 2024-09-13T11:32:45.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231353+0000", 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001539503 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232188+0000", 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083500600000000001 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232284+0000", 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6645999999999996e-05 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232325+0000", 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0071999999999999e-05 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232373+0000", 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8498999999999998e-05 2024-09-13T11:32:45.708 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:266 4.4 4:2451796c:::smithi20438004-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.444385753999999, 2024-09-13T11:32:45.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099464859999999992, 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 266 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222325+0000", 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222330+0000", 2024-09-13T11:32:45.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999997e-06 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222341+0000", 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0528e-05 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222347+0000", 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6409999999999996e-06 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222406+0000", 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8449000000000002e-05 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222487+0000", 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1167000000000004e-05 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:32:45.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230197+0000", 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077105120000000001 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230937+0000", 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000740159 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231296+0000", 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035862399999999999 2024-09-13T11:32:45.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232186+0000", 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089002899999999997 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232234+0000", 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7414999999999999e-05 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232271+0000", 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7323000000000002e-05 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.716 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:283 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.433257618999999, 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011861159, 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.717 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 283 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:32:45.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233453+0000", 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233457+0000", 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1150000000000004e-06 2024-09-13T11:32:45.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233465+0000", 2024-09-13T11:32:45.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-09-13T11:32:45.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233471+0000", 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3450000000000002e-06 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233526+0000", 2024-09-13T11:32:45.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4790000000000002e-05 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233602+0000", 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6253000000000003e-05 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237634+0000", 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040318580000000001 2024-09-13T11:32:45.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238163+0000", 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052849899999999998 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242711+0000", 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045478530000000001 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245235+0000", 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025239160000000002 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245286+0000", 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1739e-05 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245314+0000", 2024-09-13T11:32:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7280999999999999e-05 2024-09-13T11:32:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:284 4.1 4:8b052c81:::smithi20438004-35:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:32:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.432907607000001, 2024-09-13T11:32:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011653294, 2024-09-13T11:32:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 284 2024-09-13T11:32:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:32:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233803+0000", 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4600000000000003e-07 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233808+0000", 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:32:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233817+0000", 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9870000000000005e-06 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233824+0000", 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8290000000000002e-06 2024-09-13T11:32:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233874+0000", 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9877999999999999e-05 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233943+0000", 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238024+0000", 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040810350000000002 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238440+0000", 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041507799999999999 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242785+0000", 2024-09-13T11:32:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043457890000000001 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245363+0000", 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025773359999999999 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245416+0000", 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3156999999999999e-05 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245456+0000", 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0000000000000003e-05 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:286 4.5 4:a9b0c9bd:::smithi20438004-37:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.431925845999999, 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013081485, 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 286 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234785+0000", 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234790+0000", 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.651e-06 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234800+0000", 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234807+0000", 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.861e-06 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234868+0000", 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0582999999999999e-05 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234967+0000", 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:32:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054743730000000003 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241205+0000", 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076331700000000001 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241351+0000", 2024-09-13T11:32:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619399999999999 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247766+0000", 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064151290000000003 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247825+0000", 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8739999999999997e-05 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247866+0000", 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747000000000001e-05 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:290 4.1 4:8d6fdfdb:::smithi20438004-41:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.427371697000002, 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014414148999999999, 2024-09-13T11:32:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 290 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239339+0000", 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239344+0000", 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2290000000000002e-06 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239354+0000", 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239361+0000", 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.092e-06 2024-09-13T11:32:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239406+0000", 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126000000000001e-05 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239452+0000", 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6192000000000001e-05 2024-09-13T11:32:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248478+0000", 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090262209999999992 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249309+0000", 2024-09-13T11:32:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083057300000000003 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253400+0000", 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040908760000000002 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253662+0000", 2024-09-13T11:32:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026229500000000001 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253715+0000", 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2836e-05 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253753+0000", 2024-09-13T11:32:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7526e-05 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:292 4.1 4:84fa1f64:::smithi20438004-43:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.426948484, 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010951834000000001, 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 292 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8800000000000002e-07 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239765+0000", 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239773+0000", 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8229999999999995e-06 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239779+0000", 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9900000000000002e-06 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239831+0000", 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2326e-05 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239910+0000", 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8623999999999998e-05 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245574+0000", 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005663927 2024-09-13T11:32:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.246099+0000", 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052478300000000002 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247963+0000", 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001864166 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250625+0000", 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002662228 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250676+0000", 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0989000000000003e-05 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250713+0000", 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7360000000000001e-05 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:317 4.1 4:8c30e60e:::smithi20438004-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.415824578999999, 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013134860999999999, 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 317 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250892+0000", 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250900+0000", 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250908+0000", 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.337e-06 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250967+0000", 2024-09-13T11:32:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8201000000000001e-05 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251045+0000", 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299000000000004e-05 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258769+0000", 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007724323 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259599+0000", 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082958200000000004 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260992+0000", 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00139291 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263918+0000", 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029263190000000001 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263971+0000", 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2670999999999998e-05 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264020+0000", 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:330 4.3 4:c7f26bfe:::smithi20438004-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.414601764, 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011943621, 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 330 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252109+0000", 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252116+0000", 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9199999999999998e-06 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252129+0000", 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2768e-05 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252138+0000", 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5939999999999994e-06 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252205+0000", 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:32:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252280+0000", 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4930000000000003e-05 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259484+0000", 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00720434 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259994+0000", 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051054200000000001 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260333+0000", 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033855000000000002 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263947+0000", 2024-09-13T11:32:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00361452 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264015+0000", 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7155000000000003e-05 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264052+0000", 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305000000000003e-05 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:333 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.414311567999999, 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011814712, 2024-09-13T11:32:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 333 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252406+0000", 2024-09-13T11:32:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:32:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252416+0000", 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-05 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252423+0000", 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252486+0000", 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2805999999999995e-05 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252540+0000", 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3897000000000003e-05 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259138+0000", 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065977429999999997 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259785+0000", 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064727700000000003 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264056+0000", 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042706710000000002 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264152+0000", 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5877000000000004e-05 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264180+0000", 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8674999999999999e-05 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264213+0000", 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3015000000000002e-05 2024-09-13T11:32:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:433 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.389760720999998, 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098551839999999995, 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-13T11:32:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276950+0000", 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999995e-07 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276955+0000", 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1059999999999999e-06 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276965+0000", 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276977+0000", 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1518000000000001e-05 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277035+0000", 2024-09-13T11:32:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8133000000000001e-05 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277125+0000", 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0148999999999994e-05 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282235+0000", 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005109461 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282725+0000", 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049045700000000002 2024-09-13T11:32:45.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286634+0000", 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039087929999999998 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286707+0000", 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2692000000000001e-05 2024-09-13T11:32:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286772+0000", 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5828999999999996e-05 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286805+0000", 2024-09-13T11:32:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2134000000000001e-05 2024-09-13T11:32:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:791 4.5 4:a5f8f9e2:::smithi20438004-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.966877811, 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019917450999999999, 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 791 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:32:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699834+0000", 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699840+0000", 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5989999999999998e-06 2024-09-13T11:32:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699852+0000", 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699861+0000", 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5259999999999993e-06 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699949+0000", 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8083000000000004e-05 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699953+0000", 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3869999999999997e-06 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701189+0000", 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012354740000000001 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701250+0000", 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1733e-05 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.719750+0000", 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018499504 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1374 4.1 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.5868106829999995, 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011338608999999999, 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1374 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:32:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:32:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079900+0000", 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-07 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079901+0000", 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079905+0000", 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2039999999999999e-06 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079908+0000", 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8200000000000001e-06 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081697+0000", 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017893080000000001 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081802+0000", 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105231 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.091238+0000", 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094355819999999997 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1454 4.1 4:84fa1f64:::smithi20438004-43:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.8307638399999995, 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025598754000000001, 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1454 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835947+0000", 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8500000000000002e-07 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835948+0000", 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835952+0000", 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4259999999999999e-06 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835954+0000", 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8550000000000002e-06 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836019+0000", 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4416000000000001e-05 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836271+0000", 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025171299999999998 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.861545+0000", 2024-09-13T11:32:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025274496 2024-09-13T11:32:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:45.810 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:45.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:32:45.814 INFO:tasks.rados.rados.0.smithi204.stdout:912: done (12 left) 2024-09-13T11:32:45.814 INFO:tasks.rados.rados.0.smithi204.stdout:913: done (11 left) 2024-09-13T11:32:45.814 INFO:tasks.rados.rados.0.smithi204.stdout:914: done (10 left) 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:922: read oid 7 snap -1 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:922: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:923: read oid 3 snap -1 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:923: expect deleted 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:924: write (excl) oid 31 current snap is 0 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:924: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:924: writing smithi20438004-31 from 403401 to 858362 tid 1 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:924: writing smithi20438004-31 from 1584925 to 2090964 tid 2 2024-09-13T11:32:45.815 INFO:tasks.rados.rados.0.smithi204.stdout:925: read oid 12 snap -1 2024-09-13T11:32:45.816 INFO:tasks.rados.rados.0.smithi204.stdout:925: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-13T11:32:45.816 INFO:tasks.rados.rados.0.smithi204.stdout:926: tier_promote oid 45 2024-09-13T11:32:45.816 INFO:tasks.rados.rados.0.smithi204.stdout:916: expect (ObjNum 103 snap 0 seq_num 103) 2024-09-13T11:32:45.816 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:45.810+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:45.898 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:45.912 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:45.911+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:45.949 INFO:tasks.rados.rados.0.smithi204.stdout:917: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-13T11:32:45.950 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 138289 2024-09-13T11:32:45.956 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:45.956 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:45.957 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:45.957 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:45.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:32:46.001 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:46.013 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:46.011+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:46.054 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:921: got (22) Invalid argument 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:919: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:919: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:919: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:919: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:919: oid 5 updating version 0 to 47 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:919: oid 5 version 47 is already newer than 46 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:919: oid 5 version 47 is already newer than 45 2024-09-13T11:32:46.107 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 47 (ObjNum 142 snap 0 seq_num 142) dirty exists 2024-09-13T11:32:46.108 INFO:tasks.rados.rados.0.smithi204.stdout:919: left oid 5 (ObjNum 142 snap 0 seq_num 142) 2024-09-13T11:32:46.108 INFO:tasks.rados.rados.0.smithi204.stdout:916: done (14 left) 2024-09-13T11:32:46.108 INFO:tasks.rados.rados.0.smithi204.stdout:917: done (13 left) 2024-09-13T11:32:46.108 INFO:tasks.rados.rados.0.smithi204.stdout:919: done (12 left) 2024-09-13T11:32:46.108 INFO:tasks.rados.rados.0.smithi204.stdout:921: done (11 left) 2024-09-13T11:32:46.108 INFO:tasks.rados.rados.0.smithi204.stdout:927: tier_promote oid 9 2024-09-13T11:32:46.108 INFO:tasks.rados.rados.0.smithi204.stdout:922: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-13T11:32:46.108 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:46.109 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:46.109 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:46.109 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:46.109 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:46.113 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:46.112+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:46.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:32:46.176 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:46.214 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:46.213+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:46.217 INFO:tasks.rados.rados.0.smithi204.stdout:924: finishing write tid 1 to smithi20438004-31 2024-09-13T11:32:46.217 INFO:tasks.rados.rados.0.smithi204.stdout:925: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-13T11:32:46.260 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:46.260 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.915336510000003, 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079674329999999995, 2024-09-13T11:32:46.261 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4344", 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343169+0000", 2024-09-13T11:32:46.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343177+0000", 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4819999999999994e-06 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343189+0000", 2024-09-13T11:32:46.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2306999999999999e-05 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343199+0000", 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343279+0000", 2024-09-13T11:32:46.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0025999999999996e-05 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343289+0000", 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0518e-05 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348428+0000", 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051380779999999999 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348510+0000", 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2226999999999997e-05 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348598+0000", 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8722999999999997e-05 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:46.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349014+0000", 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041537900000000001 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349279+0000", 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026517700000000001 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351081+0000", 2024-09-13T11:32:46.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001802292 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351110+0000", 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8651000000000001e-05 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351132+0000", 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069000000000002e-05 2024-09-13T11:32:46.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1 4.e e38/31 4:76a15ccf:::smithi20438004-1:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.456281+0000", 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.802220180999999, 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019660440000000001, 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.456281+0000", 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.456281+0000", 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.456281+0000", 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.67e-07 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.457016+0000", 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073526899999999998 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.457026+0000", 2024-09-13T11:32:46.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9529999999999994e-06 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.457038+0000", 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.457107+0000", 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9377999999999999e-05 2024-09-13T11:32:46.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.457128+0000", 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0749000000000001e-05 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.458203+0000", 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010750860000000001 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.458247+0000", 2024-09-13T11:32:46.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3807000000000003e-05 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:21 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'4, mlcod=38'2)", 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.745353458, 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002707297, 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.275 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513149+0000", 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.322e-06 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514452+0000", 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303153 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514468+0000", 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6045999999999998e-05 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514478+0000", 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4960000000000002e-06 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514531+0000", 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3442000000000001e-05 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514546+0000", 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5211999999999999e-05 2024-09-13T11:32:46.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515821+0000", 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001274854 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515855+0000", 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3772000000000001e-05 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.279 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:53 4.2 e38/31 4:4ba9f189:::smithi20438004-14:head v 38'4, mlcod=38'2)", 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.608738+0000", 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.649762789, 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019471899999999999, 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.608738+0000", 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.608738+0000", 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.608741+0000", 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1490000000000001e-06 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.609304+0000", 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056309999999999997 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.609322+0000", 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8267999999999999e-05 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.609330+0000", 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.609387+0000", 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7252e-05 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.609412+0000", 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4655e-05 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.610651+0000", 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012396550000000001 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.610686+0000", 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4051999999999999e-05 2024-09-13T11:32:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.237126960000001, 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002340637, 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021376+0000", 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.013e-06 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022454+0000", 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010781110000000001 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022469+0000", 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4199000000000001e-05 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022477+0000", 2024-09-13T11:32:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022532+0000", 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5636000000000001e-05 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022548+0000", 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5769000000000001e-05 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023686+0000", 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011379929999999999 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023715+0000", 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8659000000000001e-05 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:182 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'8, mlcod=38'6)", 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.231061286999999, 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022898689999999999, 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027441+0000", 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7599999999999996e-07 2024-09-13T11:32:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028556+0000", 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011154439999999999 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028566+0000", 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8269999999999993e-06 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028571+0000", 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8550000000000001e-06 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028614+0000", 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028629+0000", 2024-09-13T11:32:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029703+0000", 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001073898 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029730+0000", 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6993999999999999e-05 2024-09-13T11:32:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:189 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'10, mlcod=38'8)", 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.208639984000001, 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020004459999999999, 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049862+0000", 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2999999999999999e-07 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050772+0000", 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091040000000000001 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050787+0000", 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4021e-05 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050793+0000", 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6959999999999999e-06 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050843+0000", 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0148999999999997e-05 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050860+0000", 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6665999999999999e-05 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.051831+0000", 2024-09-13T11:32:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097091600000000005 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.051862+0000", 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0667999999999998e-05 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:190 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'11, mlcod=38'9)", 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.201233557999998, 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022963139999999998, 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:32:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:32:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058460+0000", 2024-09-13T11:32:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192238 2024-09-13T11:32:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058470+0000", 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058475+0000", 2024-09-13T11:32:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5399999999999997e-06 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058514+0000", 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8887999999999998e-05 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058526+0000", 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1683e-05 2024-09-13T11:32:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059537+0000", 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001011907 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059564+0000", 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.864050250999998, 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048595679999999999, 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394452+0000", 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397390+0000", 2024-09-13T11:32:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002937977 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397401+0000", 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1172000000000001e-05 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397407+0000", 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397452+0000", 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4705e-05 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397467+0000", 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5622000000000001e-05 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399273+0000", 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001805823 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399310+0000", 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7512999999999998e-05 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1363 4.2 e43/31 4:4ba9f189:::smithi20438004-14:head v 43'50, mlcod=43'48)", 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.212917817999999, 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021379849999999998, 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045585+0000", 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.604e-06 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045958+0000", 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000372672 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045978+0000", 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0262999999999999e-05 2024-09-13T11:32:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045991+0000", 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2811e-05 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046079+0000", 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8667999999999999e-05 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046103+0000", 2024-09-13T11:32:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4131e-05 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047664+0000", 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001560098 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047721+0000", 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7738000000000002e-05 2024-09-13T11:32:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1410 4.4 e43/41 4:2451796c:::smithi20438004-17:head v 43'44, mlcod=43'42)", 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.12994531, 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022888489999999999, 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128557+0000", 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3939999999999999e-06 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129014+0000", 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045637100000000001 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129032+0000", 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8685000000000002e-05 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129042+0000", 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129100+0000", 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8238e-05 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129119+0000", 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8258e-05 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130809+0000", 2024-09-13T11:32:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001690045 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130845+0000", 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5991000000000002e-05 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:32:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.2273554040000008, 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071247150000000002, 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031147+0000", 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.579e-06 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036431+0000", 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00528394 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036449+0000", 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7329e-05 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036457+0000", 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036529+0000", 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1506000000000003e-05 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036551+0000", 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2154000000000002e-05 2024-09-13T11:32:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038234+0000", 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001683192 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038270+0000", 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6408000000000001e-05 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "pg_update_log_missing(4.e epoch 44/31 rep_tid 48 entries 44'34 (0'0) error 4:7984cf84:::smithi20438004-46:head by client.4631.0:1458 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 43'31 roll_forward_to 43'31)", 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.036496+0000", 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.2220050459999996, 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018347070000000001, 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036496+0000", 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036496+0000", 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036497+0000", 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8999999999999996e-07 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036507+0000", 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9809999999999998e-06 2024-09-13T11:32:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036527+0000", 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0247000000000001e-05 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036538+0000", 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1095e-05 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.037534+0000", 2024-09-13T11:32:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00099557199999999991 2024-09-13T11:32:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038331+0000", 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00079712200000000004 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1468 4.4 e44/41 4:2b094039:::smithi20438004-40:head v 44'48, mlcod=43'46)", 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.1895064899999994, 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020495919999999998, 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068996+0000", 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4649999999999999e-06 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069273+0000", 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027673599999999997 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069292+0000", 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9202999999999999e-05 2024-09-13T11:32:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069302+0000", 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069367+0000", 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5112999999999998e-05 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069393+0000", 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5867000000000002e-05 2024-09-13T11:32:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.070994+0000", 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016011980000000001 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.071044+0000", 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0421999999999999e-05 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1679 4.4 4.9c0290d4 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.1992976999999998, 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47635528100000002, 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1679 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059205+0000", 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.812e-06 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059211+0000", 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1469999999999998e-06 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059222+0000", 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0560000000000001e-05 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059233+0000", 2024-09-13T11:32:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0591000000000001e-05 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059286+0000", 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519e-05 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059293+0000", 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4690000000000002e-06 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535521+0000", 2024-09-13T11:32:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47622830100000002 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535559+0000", 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7882000000000001e-05 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.85605007, 2024-09-13T11:32:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002436952, 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402453+0000", 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.632e-06 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403268+0000", 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081553400000000003 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403289+0000", 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0353e-05 2024-09-13T11:32:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403298+0000", 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4760000000000004e-06 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403369+0000", 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0858000000000005e-05 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403397+0000", 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8518e-05 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404841+0000", 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001443228 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404888+0000", 2024-09-13T11:32:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7352999999999999e-05 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.85052367399999995, 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003287178, 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407979+0000", 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409585+0000", 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016060479999999999 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409598+0000", 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3027000000000001e-05 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409606+0000", 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2190000000000004e-06 2024-09-13T11:32:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409676+0000", 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9791999999999999e-05 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409707+0000", 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1692000000000001e-05 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411223+0000", 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001515705 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411265+0000", 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1649000000000001e-05 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1703 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'56, mlcod=44'53)", 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.409624+0000", 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.84887689200000005, 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019595400000000001, 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409624+0000", 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409624+0000", 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409625+0000", 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4200000000000002e-07 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409628+0000", 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3079999999999999e-06 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409633+0000", 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8269999999999997e-06 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409636+0000", 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1750000000000001e-06 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.410683+0000", 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010475370000000001 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.410704+0000", 2024-09-13T11:32:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.056e-05 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411554+0000", 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084953100000000005 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411584+0000", 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0260000000000002e-05 2024-09-13T11:32:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1705 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'62, mlcod=0'0)", 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.842919274, 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003138161, 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:32:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415583+0000", 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9699999999999995e-07 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416900+0000", 2024-09-13T11:32:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001317251 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416926+0000", 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5622e-05 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416942+0000", 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6741000000000001e-05 2024-09-13T11:32:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417014+0000", 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1298999999999996e-05 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417042+0000", 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.851e-05 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418673+0000", 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016310949999999999 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418720+0000", 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6845999999999998e-05 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1706 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'63, mlcod=45'61)", 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.83758707700000001, 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003090597, 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420915+0000", 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-06 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422306+0000", 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001390753 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422317+0000", 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422322+0000", 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4360000000000001e-06 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422379+0000", 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7003000000000003e-05 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422405+0000", 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6424e-05 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.423958+0000", 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015526680000000001 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.424005+0000", 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6619000000000002e-05 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.368 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:46.369 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 665104 2024-09-13T11:32:46.369 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 176352 2024-09-13T11:32:46.370 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:32:46.370 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:46.368+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:46.380 INFO:tasks.rados.rados.0.smithi204.stdout:924: finishing write tid 2 to smithi20438004-31 2024-09-13T11:32:46.380 INFO:tasks.rados.rados.0.smithi204.stdout:924: finishing write tid 3 to smithi20438004-31 2024-09-13T11:32:46.380 INFO:tasks.rados.rados.0.smithi204.stdout:922: done (11 left) 2024-09-13T11:32:46.380 INFO:tasks.rados.rados.0.smithi204.stdout:923: done (10 left) 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:925: done (9 left) 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:928: write (excl) oid 8 current snap is 0 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:924: finishing write tid 4 to smithi20438004-31 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:924: oid 31 updating version 0 to 37 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:924: oid 31 version 37 is already newer than 36 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:924: oid 31 updating version 37 to 38 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 38 (ObjNum 143 snap 0 seq_num 143) dirty exists 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:924: left oid 31 (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:32:46.381 INFO:tasks.rados.rados.0.smithi204.stdout:928: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-13T11:32:46.385 INFO:tasks.rados.rados.0.smithi204.stdout:928: writing smithi20438004-8 from 420208 to 950418 tid 1 2024-09-13T11:32:46.394 INFO:tasks.rados.rados.0.smithi204.stdout:928: writing smithi20438004-8 from 1527054 to 2293104 tid 2 2024-09-13T11:32:46.394 INFO:tasks.rados.rados.0.smithi204.stdout:924: done (9 left) 2024-09-13T11:32:46.394 INFO:tasks.rados.rados.0.smithi204.stdout:929: set_chunk oid 33 target oid 2024-09-13T11:32:46.394 INFO:tasks.rados.rados.0.smithi204.stdout:929: set_chunk oid 33 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:46.394 INFO:tasks.rados.rados.0.smithi204.stdout:927: got (0) Success 2024-09-13T11:32:46.394 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 32 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-13T11:32:46.394 INFO:tasks.rados.rados.0.smithi204.stdout:907: expect (ObjNum 127 snap 0 seq_num 127) 2024-09-13T11:32:46.419 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 531951 2024-09-13T11:32:46.464 INFO:tasks.rados.rados.0.smithi204.stdout:926: got (0) Success 2024-09-13T11:32:46.465 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 24 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-13T11:32:46.465 INFO:tasks.rados.rados.0.smithi204.stdout:899: got (0) Success 2024-09-13T11:32:46.465 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 0 (ObjNum 120 snap 0 seq_num 120) dirty exists 2024-09-13T11:32:46.465 INFO:tasks.rados.rados.0.smithi204.stdout:918: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:46.468 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1603160 2024-09-13T11:32:46.469 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:46.468+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:46.484 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:46.514 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:46.514 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:46.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:32:46.543 INFO:tasks.rados.rados.0.smithi204.stdout:908: got (0) Success 2024-09-13T11:32:46.543 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-13T11:32:46.543 INFO:tasks.rados.rados.0.smithi204.stdout:915: finishing write tid 1 to smithi20438004-39 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:915: finishing write tid 2 to smithi20438004-39 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:915: finishing write tid 3 to smithi20438004-39 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:915: finishing write tid 4 to smithi20438004-39 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:915: oid 39 updating version 0 to 68 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:915: oid 39 updating version 68 to 69 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:915: oid 39 version 69 is already newer than 67 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 69 (ObjNum 141 snap 0 seq_num 141) dirty exists 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:915: left oid 39 (ObjNum 141 snap 0 seq_num 141) 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:920: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:46.544 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:46.570 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:46.569+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:46.597 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:46.658 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1042153 2024-09-13T11:32:46.664 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:46.664 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:46.664 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:46.664 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:46.664 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:46.671 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:46.669+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:46.672 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:928: finishing write tid 1 to smithi20438004-8 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:929: finishing set_chunk to oid 33 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:929: got expected ENOENT (src dne) 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:928: finishing write tid 2 to smithi20438004-8 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:928: finishing write tid 3 to smithi20438004-8 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:928: finishing write tid 4 to smithi20438004-8 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:928: oid 8 updating version 0 to 16 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:928: oid 8 updating version 16 to 17 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:928: oid 8 version 17 is already newer than 15 2024-09-13T11:32:46.700 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 17 (ObjNum 144 snap 0 seq_num 144) dirty exists 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:928: left oid 8 (ObjNum 144 snap 0 seq_num 144) 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:930: read oid 43 snap -1 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:930: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:899: done (10 left) 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:907: done (9 left) 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:908: done (8 left) 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:915: done (7 left) 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:918: done (6 left) 2024-09-13T11:32:46.701 INFO:tasks.rados.rados.0.smithi204.stdout:920: done (5 left) 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:926: done (4 left) 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:927: done (3 left) 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:928: done (2 left) 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:929: done (1 left) 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:931: read oid 23 snap -1 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:931: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:932: read oid 1 snap -1 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:932: expect (ObjNum 103 snap 0 seq_num 103) 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:933: write (excl) oid 41 current snap is 0 2024-09-13T11:32:46.702 INFO:tasks.rados.rados.0.smithi204.stdout:933: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-13T11:32:46.705 INFO:tasks.rados.rados.0.smithi204.stdout:933: writing smithi20438004-41 from 437015 to 1158827 tid 1 2024-09-13T11:32:46.712 INFO:tasks.rados.rados.0.smithi204.stdout:933: writing smithi20438004-41 from 1701889 to 2211597 tid 2 2024-09-13T11:32:46.712 INFO:tasks.rados.rados.0.smithi204.stdout:934: delete oid 20 current snap is 0 2024-09-13T11:32:46.712 INFO:tasks.rados.rados.0.smithi204.stdout:931: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-13T11:32:46.772 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:46.771+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:206 3.1 e38/30 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:32:46.817 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.722843563000001, 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012415890000000001, 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092192+0000", 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.63e-07 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100358+0000", 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081664059999999993 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100384+0000", 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5287e-05 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100401+0000", 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7744000000000002e-05 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100478+0000", 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6769999999999999e-05 2024-09-13T11:32:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100508+0000", 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0134e-05 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104514+0000", 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040060590000000002 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104607+0000", 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3027000000000002e-05 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:201 3.1 e38/30 3:b265d015:::smithi20438004-2 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.714598224, 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012387119, 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100438+0000", 2024-09-13T11:32:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1000000000000001e-06 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108964+0000", 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085255959999999999 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108984+0000", 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0466e-05 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108996+0000", 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2149e-05 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109097+0000", 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010042799999999999 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109121+0000", 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.455e-05 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112773+0000", 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036519780000000002 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112824+0000", 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0852e-05 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.705976753000002, 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096300529999999995, 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109060+0000", 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116086+0000", 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070260239999999996 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116100+0000", 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4209999999999999e-05 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116109+0000", 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6569999999999992e-06 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116160+0000", 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0470000000000003e-05 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116191+0000", 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1494e-05 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118639+0000", 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024481030000000001 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118688+0000", 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8866999999999998e-05 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.698904958, 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010922463, 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116131+0000", 2024-09-13T11:32:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6899999999999998e-07 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124407+0000", 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082764919999999999 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124422+0000", 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5404999999999999e-05 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124432+0000", 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8910000000000007e-06 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124485+0000", 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2750999999999997e-05 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124508+0000", 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3076999999999999e-05 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127004+0000", 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024962930000000001 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127053+0000", 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8084999999999998e-05 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:32:46.835 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.68524086, 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091903760000000001, 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129795+0000", 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8700000000000001e-07 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135325+0000", 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055302190000000003 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135340+0000", 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135351+0000", 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1358999999999999e-05 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135418+0000", 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000002e-05 2024-09-13T11:32:46.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135447+0000", 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8415000000000002e-05 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138936+0000", 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034896800000000002 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138985+0000", 2024-09-13T11:32:46.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8338999999999999e-05 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.675045787999998, 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010981263, 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139990+0000", 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2e-07 2024-09-13T11:32:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147357+0000", 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073669950000000003 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147376+0000", 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9453000000000002e-05 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147388+0000", 2024-09-13T11:32:46.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1527e-05 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147472+0000", 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4630999999999997e-05 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147493+0000", 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0687e-05 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150899+0000", 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034054739999999999 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150971+0000", 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1976000000000003e-05 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.844 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.667594780000002, 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015742025999999999, 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147441+0000", 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7700000000000004e-07 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157019+0000", 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095782090000000007 2024-09-13T11:32:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157057+0000", 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157072+0000", 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5082999999999999e-05 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157157+0000", 2024-09-13T11:32:46.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5065e-05 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157188+0000", 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0920000000000002e-05 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163133+0000", 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059446890000000004 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163182+0000", 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9564999999999997e-05 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.657522692000001, 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015556076, 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.849 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157514+0000", 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167722+0000", 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010208016 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167740+0000", 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7668000000000001e-05 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167754+0000", 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4647999999999999e-05 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167831+0000", 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7229000000000002e-05 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167855+0000", 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3467000000000001e-05 2024-09-13T11:32:46.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173001+0000", 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051458160000000001 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173068+0000", 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7720999999999998e-05 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.647254199999999, 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011370243, 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3499999999999996e-07 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175562+0000", 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077811060000000003 2024-09-13T11:32:46.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175578+0000", 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175587+0000", 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175668+0000", 2024-09-13T11:32:46.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1224999999999996e-05 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175695+0000", 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6505000000000001e-05 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179075+0000", 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003380508 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179151+0000", 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5927000000000007e-05 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.639427539, 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017514140000000001, 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.858 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188284+0000", 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012675702 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188299+0000", 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5122e-05 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188308+0000", 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9819999999999997e-06 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188367+0000", 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8774999999999999e-05 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188387+0000", 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0649000000000002e-05 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193056+0000", 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046684830000000002 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193122+0000", 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5790999999999994e-05 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.862 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.626702560999998, 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015855429000000001, 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188333+0000", 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-07 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199600+0000", 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011266717000000001 2024-09-13T11:32:46.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199613+0000", 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3682e-05 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199620+0000", 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1339999999999998e-06 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199676+0000", 2024-09-13T11:32:46.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5689000000000002e-05 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199708+0000", 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2082999999999998e-05 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204123+0000", 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004414734 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204188+0000", 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.615394024, 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092010560000000009, 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.867 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199642+0000", 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9599999999999999e-07 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205539+0000", 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058972290000000004 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205553+0000", 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4577e-05 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205564+0000", 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0254e-05 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205630+0000", 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6057999999999997e-05 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205658+0000", 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7979000000000001e-05 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208792+0000", 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031342499999999999 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208842+0000", 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0213000000000002e-05 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.871 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:245 3.1 e38/30 3:adf25eb4:::smithi20438004-46 low_tier:head v 38'15, mlcod=38'11)", 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.609447695, 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011146911000000001, 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205588+0000", 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213344+0000", 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077554440000000002 2024-09-13T11:32:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213359+0000", 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5455999999999999e-05 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213369+0000", 2024-09-13T11:32:46.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9890000000000004e-06 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213453+0000", 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4153999999999994e-05 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213479+0000", 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5831999999999999e-05 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216694+0000", 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032149990000000001 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216734+0000", 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0102999999999999e-05 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:260 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:46.876 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.595676674, 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011239344, 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 260 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219360+0000", 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219366+0000", 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7179999999999999e-06 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219376+0000", 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9990000000000003e-06 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219382+0000", 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8150000000000002e-06 2024-09-13T11:32:46.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219428+0000", 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5751999999999998e-05 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4119000000000001e-05 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227887+0000", 2024-09-13T11:32:46.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083953989999999996 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228998+0000", 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001110764 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229290+0000", 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029145400000000002 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230489+0000", 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001199786 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230548+0000", 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8900999999999997e-05 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230598+0000", 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9391999999999997e-05 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.882 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:249 3.1 e38/30 3:81faf980:::smithi20438004-50 low_tier:head v 38'21, mlcod=38'18)", 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.592400538, 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091134460000000007, 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.68e-07 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227843+0000", 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005207494 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227853+0000", 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117e-05 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227860+0000", 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4880000000000001e-06 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227912+0000", 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1900000000000001e-05 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227933+0000", 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0474000000000001e-05 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231709+0000", 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037762759999999999 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231748+0000", 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9029000000000002e-05 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:278 4.2 4:4d4c8512:::smithi20438004-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.584040938000001, 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091148359999999994, 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.888 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230995+0000", 2024-09-13T11:32:46.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4200000000000004e-07 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231000+0000", 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3290000000000001e-06 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231017+0000", 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6242000000000001e-05 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231025+0000", 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231094+0000", 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9554999999999994e-05 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231171+0000", 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7082999999999994e-05 2024-09-13T11:32:46.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236799+0000", 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056282260000000001 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237539+0000", 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073930700000000003 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237574+0000", 2024-09-13T11:32:46.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5191999999999998e-05 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240013+0000", 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002439185 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240070+0000", 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6650999999999997e-05 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240109+0000", 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9131999999999997e-05 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:287 4.2 4:4297cb20:::smithi20438004-38:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.580105465999999, 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013317569, 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.894 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 287 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234931+0000", 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.192e-06 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234936+0000", 2024-09-13T11:32:46.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0640000000000001e-06 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234953+0000", 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7292e-05 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234961+0000", 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235026+0000", 2024-09-13T11:32:46.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5073000000000005e-05 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235111+0000", 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4635999999999994e-05 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242149+0000", 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070384990000000001 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:46.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069074700000000002 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242936+0000", 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5730000000000007e-05 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248138+0000", 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052018330000000003 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248197+0000", 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9512000000000002e-05 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248247+0000", 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:310 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:32:46.900 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.566179097999999, 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095515259999999994, 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 310 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248857+0000", 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-06 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248863+0000", 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6529999999999999e-06 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248874+0000", 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.093e-05 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248887+0000", 2024-09-13T11:32:46.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3009e-05 2024-09-13T11:32:46.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248932+0000", 2024-09-13T11:32:46.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5445999999999998e-05 2024-09-13T11:32:46.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249025+0000", 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3098999999999998e-05 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251503+0000", 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024772629999999999 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252148+0000", 2024-09-13T11:32:46.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064507900000000001 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252433+0000", 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028559100000000002 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258304+0000", 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058703100000000001 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258361+0000", 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7658999999999998e-05 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258407+0000", 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6319999999999997e-05 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:313 4.2 4:4ba9f189:::smithi20438004-14:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:32:46.907 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.565300391000001, 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013958247999999999, 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 313 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249736+0000", 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9999999999999996e-07 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249741+0000", 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5929999999999996e-06 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249750+0000", 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7369999999999995e-06 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249757+0000", 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5360000000000004e-06 2024-09-13T11:32:46.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249806+0000", 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9018999999999999e-05 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249882+0000", 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5943000000000001e-05 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258629+0000", 2024-09-13T11:32:46.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087464589999999998 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259299+0000", 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067041799999999999 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259523+0000", 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223964 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263636+0000", 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041130539999999997 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263669+0000", 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2849999999999999e-05 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263693+0000", 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3774999999999999e-05 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.913 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1437 4.2 4:4b088bf8:::smithi20438004-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.011060004999999, 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014642921, 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:46.914 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1437 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803976+0000", 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.35e-06 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803984+0000", 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2080000000000003e-06 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803995+0000", 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:32:46.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804001+0000", 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2870000000000001e-06 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804096+0000", 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5424e-05 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:46.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804165+0000", 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8596000000000005e-05 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.818618+0000", 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452906999999999 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:46.918 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:46.919 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1960479 2024-09-13T11:32:46.920 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:32:46.921 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:46.918+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:46.950 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3171245 2024-09-13T11:32:46.972 INFO:tasks.rados.rados.0.smithi204.stdout:930: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-13T11:32:47.021 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:47.019+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:47.048 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:47.063 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:47.063 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:47.063 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:47.063 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:47.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:32:47.121 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:47.120+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:47.175 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:47.175 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:47.175 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:47.175 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:47.175 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:47.181 INFO:tasks.rados.rados.0.smithi204.stdout:933: finishing write tid 1 to smithi20438004-41 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: finishing write tid 2 to smithi20438004-41 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: finishing write tid 3 to smithi20438004-41 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: finishing write tid 4 to smithi20438004-41 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: oid 41 updating version 0 to 84 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: oid 41 updating version 84 to 85 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: oid 41 version 85 is already newer than 83 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 85 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: left oid 41 (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:930: done (4 left) 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:931: done (3 left) 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:933: done (2 left) 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:934: done (1 left) 2024-09-13T11:32:47.182 INFO:tasks.rados.rados.0.smithi204.stdout:935: read oid 28 snap -1 2024-09-13T11:32:47.183 INFO:tasks.rados.rados.0.smithi204.stdout:935: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:47.183 INFO:tasks.rados.rados.0.smithi204.stdout:936: read oid 26 snap -1 2024-09-13T11:32:47.183 INFO:tasks.rados.rados.0.smithi204.stdout:936: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-13T11:32:47.183 INFO:tasks.rados.rados.0.smithi204.stdout:937: tier_promote oid 23 2024-09-13T11:32:47.183 INFO:tasks.rados.rados.0.smithi204.stdout:938: set_chunk oid 17 target oid 2024-09-13T11:32:47.184 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:32:47.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:47.196 INFO:tasks.rados.rados.0.smithi204.stdout:938: set_chunk oid 17 offset: 330752 length: 1208832 target oid 5a39148a90a7fd3199fe6766b23c049b1ac024aaff8c5b5ab2a4a2225d870752 offset: 0 snap -1 2024-09-13T11:32:47.196 INFO:tasks.rados.rados.0.smithi204.stdout:939: read oid 24 snap -1 2024-09-13T11:32:47.196 INFO:tasks.rados.rados.0.smithi204.stdout:939: expect (ObjNum 135 snap 0 seq_num 135) 2024-09-13T11:32:47.196 INFO:tasks.rados.rados.0.smithi204.stdout:940: write (excl) oid 33 current snap is 0 2024-09-13T11:32:47.196 INFO:tasks.rados.rados.0.smithi204.stdout:940: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-13T11:32:47.201 INFO:tasks.rados.rados.0.smithi204.stdout:940: writing smithi20438004-33 from 470629 to 942939 tid 1 2024-09-13T11:32:47.206 INFO:tasks.rados.rados.0.smithi204.stdout:940: writing smithi20438004-33 from 1418853 to 1932230 tid 2 2024-09-13T11:32:47.207 INFO:tasks.rados.rados.0.smithi204.stdout:941: set_chunk oid 6 target oid 2024-09-13T11:32:47.225 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:47.224+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:47.300 INFO:tasks.rados.rados.0.smithi204.stdout:941: set_chunk oid 6 offset: 188928 length: 1265152 target oid c50ec5df10547d97c3ac15fe2309b7df782f409c8a29d8938239d4f96056cf1a offset: 0 snap -1 2024-09-13T11:32:47.300 INFO:tasks.rados.rados.0.smithi204.stdout:942: write (excl) oid 18 current snap is 0 2024-09-13T11:32:47.300 INFO:tasks.rados.rados.0.smithi204.stdout:938: finishing set_chunk to oid 17 2024-09-13T11:32:47.301 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 56 (ObjNum 139 snap 0 seq_num 139) dirty exists 2024-09-13T11:32:47.301 INFO:tasks.rados.rados.0.smithi204.stdout:942: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-13T11:32:47.306 INFO:tasks.rados.rados.0.smithi204.stdout:942: writing smithi20438004-18 from 487436 to 1034995 tid 1 2024-09-13T11:32:47.314 INFO:tasks.rados.rados.0.smithi204.stdout:942: writing smithi20438004-18 from 1760982 to 2418017 tid 2 2024-09-13T11:32:47.314 INFO:tasks.rados.rados.0.smithi204.stdout:938: done (8 left) 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:943: read oid 38 snap -1 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:943: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:940: finishing write tid 1 to smithi20438004-33 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:937: got (0) Success 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 45 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:940: finishing write tid 2 to smithi20438004-33 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:940: finishing write tid 3 to smithi20438004-33 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:937: done (8 left) 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:944: read oid 13 snap -1 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:944: expect (ObjNum 117 snap 0 seq_num 117) 2024-09-13T11:32:47.315 INFO:tasks.rados.rados.0.smithi204.stdout:945: read oid 16 snap -1 2024-09-13T11:32:47.316 INFO:tasks.rados.rados.0.smithi204.stdout:945: expect (ObjNum 121 snap 0 seq_num 121) 2024-09-13T11:32:47.316 INFO:tasks.rados.rados.0.smithi204.stdout:946: read oid 4 snap -1 2024-09-13T11:32:47.316 INFO:tasks.rados.rados.0.smithi204.stdout:946: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:47.316 INFO:tasks.rados.rados.0.smithi204.stdout:947: write (excl) oid 46 current snap is 0 2024-09-13T11:32:47.316 INFO:tasks.rados.rados.0.smithi204.stdout:947: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-13T11:32:47.320 INFO:tasks.rados.rados.0.smithi204.stdout:947: writing smithi20438004-46 from 504243 to 1127051 tid 1 2024-09-13T11:32:47.324 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:47.321+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:47.327 INFO:tasks.rados.rados.0.smithi204.stdout:947: writing smithi20438004-46 from 1819464 to 2336510 tid 2 2024-09-13T11:32:47.327 INFO:tasks.rados.rados.0.smithi204.stdout:948: read oid 49 snap -1 2024-09-13T11:32:47.327 INFO:tasks.rados.rados.0.smithi204.stdout:948: expect deleted 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:949: write (excl) oid 37 current snap is 0 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:940: finishing write tid 4 to smithi20438004-33 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:940: oid 33 updating version 0 to 34 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:940: oid 33 updating version 34 to 35 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:940: oid 33 updating version 35 to 36 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 36 (ObjNum 147 snap 0 seq_num 147) dirty exists 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:940: left oid 33 (ObjNum 147 snap 0 seq_num 147) 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:932: expect (ObjNum 103 snap 0 seq_num 103) 2024-09-13T11:32:47.328 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:47.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:32:47.333 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:47.334 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:47.334 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:47.334 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:32:47.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.957213484, 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27446322899999998, 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.335 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.336 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:32:47.336 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:32:47.336 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:32:47.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.336 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:32:47.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1100000000000005e-07 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372244+0000", 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.66e-06 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372256+0000", 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1817000000000001e-05 2024-09-13T11:32:47.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372264+0000", 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0220000000000004e-06 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372320+0000", 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6437999999999997e-05 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372329+0000", 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644121+0000", 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27179224899999999 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644271+0000", 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014990000000000001 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644421+0000", 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015035600000000001 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646446+0000", 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00202454 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646524+0000", 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8839999999999997e-05 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646598+0000", 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3862999999999994e-05 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646654+0000", 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5609999999999998e-05 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646701+0000", 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6844000000000001e-05 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:207 3.2 3:59a396eb:::smithi20438004-8 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.244816437000001, 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063519737000000007, 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.343 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 207 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7099999999999998e-07 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084636+0000", 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.559e-06 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084640+0000", 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.118e-06 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084644+0000", 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8640000000000004e-06 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084687+0000", 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2295000000000001e-05 2024-09-13T11:32:47.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084780+0000", 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2880000000000005e-05 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.106902+0000", 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022122822 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112195+0000", 2024-09-13T11:32:47.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052927 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.121161+0000", 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089661740000000004 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148052+0000", 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026890595 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148105+0000", 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2673000000000002e-05 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148154+0000", 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9786000000000001e-05 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.349 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:208 3.2 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.244797721000001, 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049409478, 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.350 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 208 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084654+0000", 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084657+0000", 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1769999999999998e-06 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084663+0000", 2024-09-13T11:32:47.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8329999999999999e-06 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084668+0000", 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6200000000000004e-06 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084884+0000", 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021599499999999999 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.085046+0000", 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016192600000000001 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.094613+0000", 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095668200000000002 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.101900+0000", 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072866160000000001 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.110249+0000", 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008349034 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133903+0000", 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023654272 2024-09-13T11:32:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133976+0000", 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2581000000000002e-05 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134063+0000", 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7169000000000005e-05 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:211 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.244756691999999, 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047621726000000003, 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.357 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.358 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.358 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.358 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 211 2024-09-13T11:32:47.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.358 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084695+0000", 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6100000000000002e-07 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084697+0000", 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2419999999999999e-06 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8160000000000001e-06 2024-09-13T11:32:47.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084706+0000", 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.313e-06 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086622+0000", 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001916212 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086720+0000", 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8077000000000003e-05 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100598+0000", 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013877195 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104958+0000", 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043607899999999998 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115608+0000", 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010650138 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132197+0000", 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016589008999999998 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132257+0000", 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9948000000000003e-05 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132316+0000", 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8625000000000002e-05 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:212 3.3 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.244739112000001, 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063179242999999996, 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.364 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 212 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7300000000000002e-07 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.255e-06 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084719+0000", 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4800000000000003e-06 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084724+0000", 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.527e-06 2024-09-13T11:32:47.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086839+0000", 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021155240000000001 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086947+0000", 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107675 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109128+0000", 2024-09-13T11:32:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022180947999999999 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.113055+0000", 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039269029999999998 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.119148+0000", 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060933810000000001 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147786+0000", 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028638395000000001 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147838+0000", 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1405e-05 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147891+0000", 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3377000000000001e-05 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.370 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:216 3.2 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.238799865000001, 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055588580999999998, 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.371 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 216 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090653+0000", 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090663+0000", 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0708e-05 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090675+0000", 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:32:47.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090688+0000", 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3193000000000001e-05 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096478+0000", 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057898150000000002 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096694+0000", 2024-09-13T11:32:47.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021582700000000001 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.103295+0000", 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006601048 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108559+0000", 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052645449999999998 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.114944+0000", 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063845389999999998 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146121+0000", 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031177449999999999 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146186+0000", 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5235999999999997e-05 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146240+0000", 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3368000000000001e-05 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:220 3.2 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.213624340999999, 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033838506999999997, 2024-09-13T11:32:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 220 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115828+0000", 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115832+0000", 2024-09-13T11:32:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.746e-06 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115842+0000", 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2010000000000005e-06 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115847+0000", 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9740000000000001e-06 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115897+0000", 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0114999999999997e-05 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116103+0000", 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020598799999999999 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135329+0000", 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019226422999999999 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141061+0000", 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057316709999999998 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.142490+0000", 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001428805 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149552+0000", 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070625610000000002 2024-09-13T11:32:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149602+0000", 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9826e-05 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149665+0000", 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3168000000000002e-05 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:226 3.3 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.195072893999999, 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.062274747999999998, 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 226 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134379+0000", 2024-09-13T11:32:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1850000000000001e-06 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134392+0000", 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2649999999999999e-05 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134404+0000", 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1615e-05 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134411+0000", 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3509999999999996e-06 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134474+0000", 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2673000000000001e-05 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134662+0000", 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018810100000000001 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149047+0000", 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014384902 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.154919+0000", 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058721379999999998 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.171898+0000", 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016979300999999999 2024-09-13T11:32:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196539+0000", 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024641252999999998 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196614+0000", 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5160999999999994e-05 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196653+0000", 2024-09-13T11:32:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8417999999999997e-05 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:228 3.3 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.188790361999999, 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038699092999999997, 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140662+0000", 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140670+0000", 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8630000000000008e-06 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140696+0000", 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5927000000000001e-05 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140708+0000", 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140804+0000", 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6137999999999993e-05 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140995+0000", 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191316 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.145999+0000", 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050033129999999997 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152237+0000", 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062383320000000001 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160262+0000", 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080251520000000007 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179164+0000", 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018901823000000002 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179296+0000", 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013243400000000001 2024-09-13T11:32:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179360+0000", 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3328000000000001e-05 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:229 3.3 3:d73fa370:::smithi20438004-30 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.188656335000001, 2024-09-13T11:32:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057728689, 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 229 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140796+0000", 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9599999999999998e-07 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140800+0000", 2024-09-13T11:32:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140809+0000", 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3669999999999997e-06 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140814+0000", 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141203+0000", 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038959299999999998 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141375+0000", 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017155499999999999 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.156365+0000", 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014989954999999999 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162876+0000", 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065109360000000002 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.187965+0000", 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025088847000000001 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198445+0000", 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010480171 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198487+0000", 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2635000000000001e-05 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198523+0000", 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5941e-05 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:238 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.17669136, 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.059471032, 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 238 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152761+0000", 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:32:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152772+0000", 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1122e-05 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152782+0000", 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5179999999999993e-06 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152790+0000", 2024-09-13T11:32:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7260000000000002e-06 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152856+0000", 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6304999999999996e-05 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153180+0000", 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000323617 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167201+0000", 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014021264 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.172985+0000", 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057843349999999998 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188702+0000", 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015716765000000001 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212106+0000", 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023403714999999999 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212169+0000", 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3224000000000003e-05 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212231+0000", 2024-09-13T11:32:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1841999999999995e-05 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:240 3.2 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.169201384000001, 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034186595, 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160251+0000", 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.562e-06 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160258+0000", 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6640000000000001e-06 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160269+0000", 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.152e-05 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160277+0000", 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4939999999999997e-06 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160327+0000", 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0268000000000001e-05 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160497+0000", 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017018900000000001 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.164999+0000", 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045015929999999999 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168431+0000", 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003431698 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.184172+0000", 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015741852000000001 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194334+0000", 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010161646999999999 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194390+0000", 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6218999999999998e-05 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194436+0000", 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5889000000000001e-05 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:242 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 18] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.160813576999999, 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053252359999999999, 2024-09-13T11:32:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 242 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168640+0000", 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168652+0000", 2024-09-13T11:32:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1909000000000001e-05 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168668+0000", 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5888000000000001e-05 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168680+0000", 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.154e-05 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169643+0000", 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096305100000000003 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000139834 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174850+0000", 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005067841 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.178949+0000", 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040991209999999998 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193069+0000", 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014119166000000001 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221801+0000", 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028732905999999999 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221852+0000", 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0958000000000002e-05 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221890+0000", 2024-09-13T11:32:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7398999999999999e-05 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:244 3.2 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.148873553000001, 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040499786000000003, 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180579+0000", 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.499e-06 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180585+0000", 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4649999999999999e-06 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180598+0000", 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180604+0000", 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5300000000000004e-06 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180669+0000", 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4859999999999999e-05 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180840+0000", 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171791 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.190029+0000", 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091888999999999998 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193312+0000", 2024-09-13T11:32:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829579999999999 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.207773+0000", 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014461039 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.220989+0000", 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013215644 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221038+0000", 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9421e-05 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221077+0000", 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8899999999999997e-05 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:247 3.2 3:4f19c0b8:::smithi20438004-48 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.143636649000001, 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040924556000000001, 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 247 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185816+0000", 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9690000000000001e-06 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185825+0000", 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6980000000000004e-06 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185842+0000", 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6492000000000001e-05 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185852+0000", 2024-09-13T11:32:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0412e-05 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185951+0000", 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9312000000000001e-05 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186223+0000", 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027177000000000001 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208403+0000", 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022179780999999999 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029604150000000001 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.223543+0000", 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012179840000000001 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226659+0000", 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031159989999999999 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226698+0000", 2024-09-13T11:32:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9020999999999998e-05 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226739+0000", 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0846999999999997e-05 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:248 3.2 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:32:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.143313568, 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035109013000000001, 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 248 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186139+0000", 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.333e-06 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186157+0000", 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8488999999999999e-05 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186170+0000", 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2737e-05 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186181+0000", 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1279e-05 2024-09-13T11:32:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186356+0000", 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174988 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186493+0000", 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137045 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:32:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.203390+0000", 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016897169 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.206673+0000", 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829510000000001 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.214284+0000", 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076100830000000001 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221141+0000", 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068573740000000003 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221198+0000", 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6916999999999997e-05 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221246+0000", 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8647999999999999e-05 2024-09-13T11:32:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.10.22:30 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.538276492000001, 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87173941200000005, 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791176+0000", 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791183+0000", 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.956e-06 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791193+0000", 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8649999999999996e-06 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791200+0000", 2024-09-13T11:32:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.148e-06 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791263+0000", 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3038000000000002e-05 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791370+0000", 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010721999999999999 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791704+0000", 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033406100000000003 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.792618+0000", 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091326999999999999 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.793087+0000", 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046985199999999999 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662893+0000", 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86980523300000001 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662914+0000", 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1319e-05 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.1.22:76 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.524628978999999, 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85814986199999999, 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804824+0000", 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5889999999999999e-06 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804830+0000", 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7139999999999996e-06 2024-09-13T11:32:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804845+0000", 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5158e-05 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804853+0000", 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9349999999999994e-06 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804910+0000", 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6501e-05 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805017+0000", 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107237 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805213+0000", 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019558100000000001 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806246+0000", 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001033225 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806260+0000", 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4072e-05 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662951+0000", 2024-09-13T11:32:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85669131700000001 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662972+0000", 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0533000000000001e-05 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:9 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.484105278000001, 2024-09-13T11:32:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81776348899999995, 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:32:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845348+0000", 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4090000000000001e-06 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845360+0000", 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1420000000000001e-05 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845389+0000", 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9453000000000001e-05 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845404+0000", 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4919000000000001e-05 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845526+0000", 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121588 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845617+0000", 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1119000000000006e-05 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845789+0000", 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017257000000000001 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846396+0000", 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060634700000000003 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846771+0000", 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037541099999999997 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663066+0000", 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81629477399999995 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663109+0000", 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3479000000000002e-05 2024-09-13T11:32:47.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:10 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.480368849000001, 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81407563599999999, 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849083+0000", 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6300000000000004e-07 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849089+0000", 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849101+0000", 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-05 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849106+0000", 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7309999999999998e-06 2024-09-13T11:32:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849154+0000", 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8077000000000001e-05 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849235+0000", 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0770000000000001e-05 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849399+0000", 2024-09-13T11:32:47.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163385 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850247+0000", 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084889399999999999 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850266+0000", 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8698e-05 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:32:47.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663146+0000", 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81287977300000003 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663158+0000", 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1928e-05 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:32:47.465 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:47.466 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:47.467 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:47.467 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:47.467 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:47.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:32:47.468 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:47.469 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:47.469 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:47.469 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:47.469 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:47.470 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:47.470 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:47.470 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:47.471 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:47.471 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:47.471 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:47.471 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:47.465+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:47.472 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:47.472 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:47.477 INFO:tasks.rados.rados.0.smithi204.stdout:936: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-13T11:32:47.486 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:47.486 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:47.502 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:47.503 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:47.518 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:47.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:47.536 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:47.536 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:47.568 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:47.566+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:47.570 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:47.570 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:47.581 INFO:tasks.rados.rados.0.smithi204.stdout:939: expect (ObjNum 135 snap 0 seq_num 135) 2024-09-13T11:32:47.587 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:47.588 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:47.588 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:47.588 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:47.591 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:47.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:32:47.668 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:47.666+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:47.669 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1751563 2024-09-13T11:32:47.679 INFO:tasks.rados.rados.0.smithi204.stdout:935: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:47.679 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:47.701 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:47.701 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:47.706 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 633380 2024-09-13T11:32:47.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:32:47.734 INFO:tasks.rados.rados.0.smithi204.stdout:941: finishing set_chunk to oid 6 2024-09-13T11:32:47.734 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 188928 length: 1265152 2024-09-13T11:32:47.734 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:47.734 INFO:tasks.rados.rados.0.smithi204.stdout:943: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-13T11:32:47.737 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 582505 2024-09-13T11:32:47.769 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:47.768+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:47.781 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1569557 2024-09-13T11:32:47.814 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:47.814 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:47.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:32:47.839 INFO:tasks.rados.rados.0.smithi204.stdout:946: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:47.861 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:47.870 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:47.869+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:47.903 INFO:tasks.rados.rados.0.smithi204.stdout:944: expect (ObjNum 117 snap 0 seq_num 117) 2024-09-13T11:32:47.926 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:47.926 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:47.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:32:47.971 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:47.969+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:48.038 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.038 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:947: finishing write tid 1 to smithi20438004-46 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:947: finishing write tid 2 to smithi20438004-46 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:947: finishing write tid 3 to smithi20438004-46 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:947: finishing write tid 4 to smithi20438004-46 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:947: oid 46 updating version 0 to 36 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:947: oid 46 updating version 36 to 38 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:947: oid 46 version 38 is already newer than 37 2024-09-13T11:32:48.069 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 38 (ObjNum 149 snap 0 seq_num 149) dirty exists 2024-09-13T11:32:48.070 INFO:tasks.rados.rados.0.smithi204.stdout:947: left oid 46 (ObjNum 149 snap 0 seq_num 149) 2024-09-13T11:32:48.070 INFO:tasks.rados.rados.0.smithi204.stdout:949: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-13T11:32:48.072 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:48.070+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:48.075 INFO:tasks.rados.rados.0.smithi204.stdout:949: writing smithi20438004-37 from 521050 to 1219107 tid 1 2024-09-13T11:32:48.083 INFO:tasks.rados.rados.0.smithi204.stdout:949: writing smithi20438004-37 from 1877946 to 2521050 tid 2 2024-09-13T11:32:48.083 INFO:tasks.rados.rados.0.smithi204.stdout:932: done (13 left) 2024-09-13T11:32:48.083 INFO:tasks.rados.rados.0.smithi204.stdout:935: done (12 left) 2024-09-13T11:32:48.083 INFO:tasks.rados.rados.0.smithi204.stdout:936: done (11 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:939: done (10 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:940: done (9 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:941: done (8 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:943: done (7 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:944: done (6 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:946: done (5 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:947: done (4 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:948: done (3 left) 2024-09-13T11:32:48.084 INFO:tasks.rados.rados.0.smithi204.stdout:950: set_chunk oid 28 target oid 2024-09-13T11:32:48.150 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.150 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:32:48.173 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:48.171+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:48.262 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.263 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.268 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:32:48.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:32:48.273 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:48.271+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:950: set_chunk oid 28 offset: 1697280 length: 441344 target oid a8e8cd0435e597bc957d6647fb12c9a68b558fe224dd545506e427cbbdbf9ff8 offset: 0 snap -1 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:951: read oid 15 snap -1 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:951: expect deleted 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:952: tier_evict oid 32 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:952: tier_evict oid 32 snap -1 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:949: finishing write tid 1 to smithi20438004-37 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:953: set_chunk oid 26 target oid 2024-09-13T11:32:48.288 INFO:tasks.rados.rados.0.smithi204.stdout:949: finishing write tid 2 to smithi20438004-37 2024-09-13T11:32:48.289 INFO:tasks.rados.rados.0.smithi204.stdout:949: finishing write tid 3 to smithi20438004-37 2024-09-13T11:32:48.289 INFO:tasks.rados.rados.0.smithi204.stdout:949: finishing write tid 4 to smithi20438004-37 2024-09-13T11:32:48.289 INFO:tasks.rados.rados.0.smithi204.stdout:949: oid 37 updating version 0 to 36 2024-09-13T11:32:48.289 INFO:tasks.rados.rados.0.smithi204.stdout:949: oid 37 version 36 is already newer than 35 2024-09-13T11:32:48.289 INFO:tasks.rados.rados.0.smithi204.stdout:949: oid 37 version 36 is already newer than 34 2024-09-13T11:32:48.289 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 36 (ObjNum 150 snap 0 seq_num 150) dirty exists 2024-09-13T11:32:48.289 INFO:tasks.rados.rados.0.smithi204.stdout:949: left oid 37 (ObjNum 150 snap 0 seq_num 150) 2024-09-13T11:32:48.294 INFO:tasks.rados.rados.0.smithi204.stdout:953: set_chunk oid 26 offset: 1832960 length: 99328 target oid 547c62bbffb2b53aaf13b7ca1f0f12bbffadd423e9993e3ecf40f8cb2d94b0c1 offset: 0 snap -1 2024-09-13T11:32:48.294 INFO:tasks.rados.rados.0.smithi204.stdout:949: done (6 left) 2024-09-13T11:32:48.294 INFO:tasks.rados.rados.0.smithi204.stdout:954: read oid 47 snap -1 2024-09-13T11:32:48.294 INFO:tasks.rados.rados.0.smithi204.stdout:954: expect (ObjNum 119 snap 0 seq_num 119) 2024-09-13T11:32:48.294 INFO:tasks.rados.rados.0.smithi204.stdout:955: write (excl) oid 17 current snap is 0 2024-09-13T11:32:48.294 INFO:tasks.rados.rados.0.smithi204.stdout:952: got (22) Invalid argument 2024-09-13T11:32:48.294 INFO:tasks.rados.rados.0.smithi204.stdout:955: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-13T11:32:48.302 INFO:tasks.rados.rados.0.smithi204.stdout:955: writing smithi20438004-17 from 537857 to 1311163 tid 1 2024-09-13T11:32:48.308 INFO:tasks.rados.rados.0.smithi204.stdout:955: writing smithi20438004-17 from 1936428 to 2457143 tid 2 2024-09-13T11:32:48.308 INFO:tasks.rados.rados.0.smithi204.stdout:952: done (7 left) 2024-09-13T11:32:48.308 INFO:tasks.rados.rados.0.smithi204.stdout:956: read oid 1 snap -1 2024-09-13T11:32:48.308 INFO:tasks.rados.rados.0.smithi204.stdout:956: expect (ObjNum 103 snap 0 seq_num 103) 2024-09-13T11:32:48.308 INFO:tasks.rados.rados.0.smithi204.stdout:957: delete oid 42 current snap is 0 2024-09-13T11:32:48.308 INFO:tasks.rados.rados.0.smithi204.stdout:945: expect (ObjNum 121 snap 0 seq_num 121) 2024-09-13T11:32:48.308 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:48.371 INFO:tasks.rados.rados.0.smithi204.stdout:942: finishing write tid 1 to smithi20438004-18 2024-09-13T11:32:48.372 INFO:tasks.rados.rados.0.smithi204.stdout:950: finishing set_chunk to oid 28 2024-09-13T11:32:48.372 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 43 (ObjNum 132 snap 0 seq_num 132) dirty exists 2024-09-13T11:32:48.372 INFO:tasks.rados.rados.0.smithi204.stdout:954: expect (ObjNum 119 snap 0 seq_num 119) 2024-09-13T11:32:48.374 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:48.373+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:48.376 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.376 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:32:48.398 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 342206 2024-09-13T11:32:48.423 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:48.451 INFO:tasks.rados.rados.0.smithi204.stdout:942: finishing write tid 2 to smithi20438004-18 2024-09-13T11:32:48.451 INFO:tasks.rados.rados.0.smithi204.stdout:942: finishing write tid 3 to smithi20438004-18 2024-09-13T11:32:48.451 INFO:tasks.rados.rados.0.smithi204.stdout:942: finishing write tid 4 to smithi20438004-18 2024-09-13T11:32:48.451 INFO:tasks.rados.rados.0.smithi204.stdout:942: oid 18 updating version 0 to 87 2024-09-13T11:32:48.451 INFO:tasks.rados.rados.0.smithi204.stdout:942: oid 18 updating version 87 to 88 2024-09-13T11:32:48.451 INFO:tasks.rados.rados.0.smithi204.stdout:942: oid 18 version 88 is already newer than 86 2024-09-13T11:32:48.451 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 88 (ObjNum 148 snap 0 seq_num 148) dirty exists 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:942: left oid 18 (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:953: finishing set_chunk to oid 26 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 20 (ObjNum 136 snap 0 seq_num 136) dirty exists 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:955: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:955: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:955: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:942: done (8 left) 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:945: done (7 left) 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:950: done (6 left) 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:951: done (5 left) 2024-09-13T11:32:48.452 INFO:tasks.rados.rados.0.smithi204.stdout:953: done (4 left) 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:954: done (3 left) 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:957: done (2 left) 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:958: tier_promote oid 23 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:955: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:955: oid 17 updating version 0 to 60 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:955: oid 17 version 60 is already newer than 59 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:955: oid 17 updating version 60 to 61 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 61 (ObjNum 151 snap 0 seq_num 151) dirty exists 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:955: left oid 17 (ObjNum 151 snap 0 seq_num 151) 2024-09-13T11:32:48.453 INFO:tasks.rados.rados.0.smithi204.stdout:956: expect (ObjNum 103 snap 0 seq_num 103) 2024-09-13T11:32:48.475 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:48.474+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:48.495 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:48.530 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.530 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.538 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:32:48.573 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2866210 2024-09-13T11:32:48.576 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:48.575+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:48.588 INFO:tasks.rados.rados.0.smithi204.stdout:958: got (0) Success 2024-09-13T11:32:48.588 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 45 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-09-13T11:32:48.588 INFO:tasks.rados.rados.0.smithi204.stdout:955: done (2 left) 2024-09-13T11:32:48.588 INFO:tasks.rados.rados.0.smithi204.stdout:956: done (1 left) 2024-09-13T11:32:48.589 INFO:tasks.rados.rados.0.smithi204.stdout:958: done (0 left) 2024-09-13T11:32:48.589 INFO:tasks.rados.rados.0.smithi204.stdout:959: set_chunk oid 32 target oid 2024-09-13T11:32:48.594 INFO:tasks.rados.rados.0.smithi204.stdout:959: set_chunk oid 32 offset: 1216512 length: 424448 target oid 7b05bc9509f64ae13fcd0991f2c66ce8638ee2500f1a50fee1217997bdac3e95 offset: 0 snap -1 2024-09-13T11:32:48.594 INFO:tasks.rados.rados.0.smithi204.stdout:960: read oid 11 snap -1 2024-09-13T11:32:48.594 INFO:tasks.rados.rados.0.smithi204.stdout:960: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-13T11:32:48.594 INFO:tasks.rados.rados.0.smithi204.stdout:961: set_chunk oid 43 target oid 2024-09-13T11:32:48.612 INFO:tasks.rados.rados.0.smithi204.stdout:961: set_chunk oid 43 offset: 236032 length: 1482752 target oid 2cbb02689b32344d6bc701d0c8f6b199a474383d9f52f69439944eadc029caee offset: 0 snap -1 2024-09-13T11:32:48.612 INFO:tasks.rados.rados.0.smithi204.stdout:962: write oid 7 current snap is 0 2024-09-13T11:32:48.612 INFO:tasks.rados.rados.0.smithi204.stdout:959: finishing set_chunk to oid 32 2024-09-13T11:32:48.612 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 1216512 length: 424448 2024-09-13T11:32:48.612 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-13T11:32:48.612 INFO:tasks.rados.rados.0.smithi204.stdout:962: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-13T11:32:48.619 INFO:tasks.rados.rados.0.smithi204.stdout:962: writing smithi20438004-7 from 571471 to 1211628 tid 1 2024-09-13T11:32:48.626 INFO:tasks.rados.rados.0.smithi204.stdout:962: writing smithi20438004-7 from 1653392 to 2177776 tid 2 2024-09-13T11:32:48.626 INFO:tasks.rados.rados.0.smithi204.stdout:959: done (3 left) 2024-09-13T11:32:48.626 INFO:tasks.rados.rados.0.smithi204.stdout:963: read oid 24 snap -1 2024-09-13T11:32:48.626 INFO:tasks.rados.rados.0.smithi204.stdout:963: expect (ObjNum 135 snap 0 seq_num 135) 2024-09-13T11:32:48.626 INFO:tasks.rados.rados.0.smithi204.stdout:964: set_chunk oid 35 target oid 2024-09-13T11:32:48.645 INFO:tasks.rados.rados.0.smithi204.stdout:964: set_chunk oid 35 offset: 1017856 length: 1737216 target oid cd1d9195765aa44641fdcdd583fd8492d280345ba2bdd5f9d8e1ee67349aecbe offset: 0 snap -1 2024-09-13T11:32:48.645 INFO:tasks.rados.rados.0.smithi204.stdout:965: write (excl) oid 8 current snap is 0 2024-09-13T11:32:48.645 INFO:tasks.rados.rados.0.smithi204.stdout:965: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-13T11:32:48.652 INFO:tasks.rados.rados.0.smithi204.stdout:965: writing smithi20438004-8 from 588278 to 1303684 tid 1 2024-09-13T11:32:48.659 INFO:tasks.rados.rados.0.smithi204.stdout:965: writing smithi20438004-8 from 1711874 to 2379916 tid 2 2024-09-13T11:32:48.659 INFO:tasks.rados.rados.0.smithi204.stdout:966: read oid 34 snap -1 2024-09-13T11:32:48.659 INFO:tasks.rados.rados.0.smithi204.stdout:966: expect (ObjNum 124 snap 0 seq_num 124) 2024-09-13T11:32:48.659 INFO:tasks.rados.rados.0.smithi204.stdout:967: write oid 26 current snap is 0 2024-09-13T11:32:48.659 INFO:tasks.rados.rados.0.smithi204.stdout:967: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-13T11:32:48.659 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.660 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.666 INFO:tasks.rados.rados.0.smithi204.stdout:967: writing smithi20438004-26 from 605085 to 1395740 tid 1 2024-09-13T11:32:48.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:32:48.673 INFO:tasks.rados.rados.0.smithi204.stdout:967: writing smithi20438004-26 from 2170356 to 2605085 tid 2 2024-09-13T11:32:48.674 INFO:tasks.rados.rados.0.smithi204.stdout:968: set_chunk oid 17 target oid 2024-09-13T11:32:48.677 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:48.675+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:48.679 INFO:tasks.rados.rados.0.smithi204.stdout:968: set_chunk oid 17 offset: 1257472 length: 412160 target oid 2afdf20452eef2c88790f496e23c06a7e2c9ec5bc753d6606e6682e3a679b833 offset: 0 snap -1 2024-09-13T11:32:48.679 INFO:tasks.rados.rados.0.smithi204.stdout:969: read oid 46 snap -1 2024-09-13T11:32:48.679 INFO:tasks.rados.rados.0.smithi204.stdout:969: expect (ObjNum 149 snap 0 seq_num 149) 2024-09-13T11:32:48.679 INFO:tasks.rados.rados.0.smithi204.stdout:970: read oid 19 snap -1 2024-09-13T11:32:48.679 INFO:tasks.rados.rados.0.smithi204.stdout:970: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:48.679 INFO:tasks.rados.rados.0.smithi204.stdout:971: read oid 22 snap -1 2024-09-13T11:32:48.680 INFO:tasks.rados.rados.0.smithi204.stdout:971: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-13T11:32:48.680 INFO:tasks.rados.rados.0.smithi204.stdout:972: write oid 38 current snap is 0 2024-09-13T11:32:48.680 INFO:tasks.rados.rados.0.smithi204.stdout:972: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-13T11:32:48.684 INFO:tasks.rados.rados.0.smithi204.stdout:972: writing smithi20438004-38 from 621892 to 1087796 tid 1 2024-09-13T11:32:48.694 INFO:tasks.rados.rados.0.smithi204.stdout:972: writing smithi20438004-38 from 1712485 to 2384196 tid 2 2024-09-13T11:32:48.694 INFO:tasks.rados.rados.0.smithi204.stdout:973: read oid 31 snap -1 2024-09-13T11:32:48.695 INFO:tasks.rados.rados.0.smithi204.stdout:973: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:32:48.695 INFO:tasks.rados.rados.0.smithi204.stdout:974: read oid 39 snap -1 2024-09-13T11:32:48.695 INFO:tasks.rados.rados.0.smithi204.stdout:974: expect (ObjNum 141 snap 0 seq_num 141) 2024-09-13T11:32:48.695 INFO:tasks.rados.rados.0.smithi204.stdout:975: read oid 40 snap -1 2024-09-13T11:32:48.695 INFO:tasks.rados.rados.0.smithi204.stdout:975: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-13T11:32:48.695 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:48.696 INFO:tasks.rados.rados.0.smithi204.stdout:960: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-13T11:32:48.728 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:48.778 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:48.777+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:48.785 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.785 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.794 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:32:48.841 INFO:tasks.rados.rados.0.smithi204.stdout:961: finishing set_chunk to oid 43 2024-09-13T11:32:48.841 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk returned error code -95 offset: 236032 length: 1482752 2024-09-13T11:32:48.841 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-13T11:32:48.841 INFO:tasks.rados.rados.0.smithi204.stdout:962: finishing write tid 1 to smithi20438004-7 2024-09-13T11:32:48.841 INFO:tasks.rados.rados.0.smithi204.stdout:963: expect (ObjNum 135 snap 0 seq_num 135) 2024-09-13T11:32:48.841 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:48.879 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:48.877+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:48.889 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1485826 2024-09-13T11:32:48.916 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1294468 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:962: finishing write tid 2 to smithi20438004-7 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:962: finishing write tid 3 to smithi20438004-7 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:962: finishing write tid 4 to smithi20438004-7 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:962: oid 7 updating version 0 to 42 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:962: oid 7 updating version 42 to 43 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:962: oid 7 updating version 43 to 44 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 44 (ObjNum 153 snap 0 seq_num 153) dirty exists 2024-09-13T11:32:48.937 INFO:tasks.rados.rados.0.smithi204.stdout:962: left oid 7 (ObjNum 153 snap 0 seq_num 153) 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:964: finishing set_chunk to oid 35 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 68 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:965: finishing write tid 1 to smithi20438004-8 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:960: done (15 left) 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:961: done (14 left) 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:962: done (13 left) 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:963: done (12 left) 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:964: done (11 left) 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:976: read oid 37 snap -1 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:976: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-13T11:32:48.938 INFO:tasks.rados.rados.0.smithi204.stdout:977: delete oid 32 current snap is 0 2024-09-13T11:32:48.939 INFO:tasks.rados.rados.0.smithi204.stdout:966: expect (ObjNum 124 snap 0 seq_num 124) 2024-09-13T11:32:48.939 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:48.939 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:48.940 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:48.949 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:32:48.970 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:49.009 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1044335 2024-09-13T11:32:49.010 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:49.008+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:49.031 INFO:tasks.rados.rados.0.smithi204.stdout:965: finishing write tid 2 to smithi20438004-8 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:965: finishing write tid 3 to smithi20438004-8 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:965: finishing write tid 4 to smithi20438004-8 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:965: oid 8 updating version 0 to 19 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:965: oid 8 version 19 is already newer than 18 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:965: oid 8 updating version 19 to 20 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 20 (ObjNum 154 snap 0 seq_num 154) dirty exists 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:965: left oid 8 (ObjNum 154 snap 0 seq_num 154) 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:967: finishing write tid 1 to smithi20438004-26 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:965: done (12 left) 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:966: done (11 left) 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:977: done (10 left) 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:978: tier_evict oid 49 2024-09-13T11:32:49.032 INFO:tasks.rados.rados.0.smithi204.stdout:978: tier_evict oid 49 snap -1 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:967: finishing write tid 2 to smithi20438004-26 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:967: finishing write tid 3 to smithi20438004-26 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:967: finishing write tid 4 to smithi20438004-26 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:967: oid 26 updating version 0 to 24 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:967: oid 26 version 24 is already newer than 22 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:967: oid 26 version 24 is already newer than 23 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 24 (ObjNum 155 snap 0 seq_num 155) dirty exists 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:967: left oid 26 (ObjNum 155 snap 0 seq_num 155) 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:969: expect (ObjNum 149 snap 0 seq_num 149) 2024-09-13T11:32:49.033 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:49.073 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.074 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.080 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:49.079+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:49.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:32:49.110 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:49.150 INFO:tasks.rados.rados.0.smithi204.stdout:970: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:49.181 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:49.179+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:49.214 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.214 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.222 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:32:49.246 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:49.281 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:49.280+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:49.303 INFO:tasks.rados.rados.0.smithi204.stdout:971: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-13T11:32:49.305 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1727342 2024-09-13T11:32:49.350 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.350 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.359 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:32:49.373 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:49.382 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:49.381+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:968: finishing set_chunk to oid 17 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 61 (ObjNum 151 snap 0 seq_num 151) dirty exists 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:972: finishing write tid 1 to smithi20438004-38 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:972: finishing write tid 2 to smithi20438004-38 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:972: finishing write tid 3 to smithi20438004-38 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:967: done (10 left) 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:968: done (9 left) 2024-09-13T11:32:49.415 INFO:tasks.rados.rados.0.smithi204.stdout:969: done (8 left) 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:970: done (7 left) 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:971: done (6 left) 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:979: read oid 3 snap -1 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:979: expect deleted 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:980: set_chunk oid 32 target oid 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:980: set_chunk oid 32 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:981: tier_evict oid 35 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:981: tier_evict oid 35 snap -1 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:972: finishing write tid 4 to smithi20438004-38 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:972: oid 38 updating version 0 to 66 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:972: oid 38 version 66 is already newer than 65 2024-09-13T11:32:49.416 INFO:tasks.rados.rados.0.smithi204.stdout:972: oid 38 updating version 66 to 67 2024-09-13T11:32:49.417 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 67 (ObjNum 156 snap 0 seq_num 156) dirty exists 2024-09-13T11:32:49.417 INFO:tasks.rados.rados.0.smithi204.stdout:972: left oid 38 (ObjNum 156 snap 0 seq_num 156) 2024-09-13T11:32:49.417 INFO:tasks.rados.rados.0.smithi204.stdout:975: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-13T11:32:49.417 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 43776 2024-09-13T11:32:49.452 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1251098 2024-09-13T11:32:49.463 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.464 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.472 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:32:49.483 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:49.482+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:49.494 INFO:tasks.rados.rados.0.smithi204.stdout:974: expect (ObjNum 141 snap 0 seq_num 141) 2024-09-13T11:32:49.518 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2002242 2024-09-13T11:32:49.562 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:49.562 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:32:49.562 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:32:49.563 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-13T11:32:49.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:32:49.573 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:49.585 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:49.584+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:49.592 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:49.597 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.597 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.607 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:32:49.620 INFO:tasks.rados.rados.0.smithi204.stdout:973: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:32:49.685 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:49.684+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:49.704 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1868019 2024-09-13T11:32:49.723 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.723 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.732 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:32:49.740 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:49.741 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:49.758 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:49.759 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:49.759 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:49.764 INFO:tasks.rados.rados.0.smithi204.stdout:976: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-13T11:32:49.764 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:49.775 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:49.775 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:49.786 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:49.785+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:49.791 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:49.791 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:49.808 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:49.809 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:49.826 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:49.826 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:49.844 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:49.844 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:49.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:49.861 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:49.868 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.868 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.877 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:32:49.878 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:49.879 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:49.880 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:32:49.880 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:32:49.888 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:49.886+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:49.891 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-13T11:32:48.935542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":202569488,"num_objects":149,"num_object_clones":0,"num_object_copies":447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1795,"num_read_kb":988281,"num_write":1370,"num_write_kb":370086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":116698501,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":26,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":83,"acting":85,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":729460,"kb_used_data":399200,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1131732620,"statfs":{"total":1159641169920,"available":1158894202880,"internally_reserved":0,"allocated":408780800,"data_stored":404998251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19137,"internal_metadata":336115007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5551171,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":102,"num_read_kb":81502,"num_write":69,"num_write_kb":14913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":98786338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001067"},"pg_stats":[{"pgid":"4.f","version":"51'69","reported_seq":195,"reported_epoch":51,"state":"peering","last_fresh":"2024-09-13T11:32:46.333793+0000","last_change":"2024-09-13T11:32:47.269910+0000","last_active":"2024-09-13T11:32:47.269910+0000","last_peered":"2024-09-13T11:32:46.333793+0000","last_clean":"2024-09-13T11:32:46.333793+0000","last_became_active":"2024-09-13T11:32:46.275133+0000","last_became_peered":"2024-09-13T11:32:46.275133+0000","last_unstale":"2024-09-13T11:32:46.333793+0000","last_undegraded":"2024-09-13T11:32:46.333793+0000","last_fullsized":"2024-09-13T11:32:46.333793+0000","mapping_epoch":52,"log_start":"45'66","ondisk_log_start":"45'66","created":31,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":66,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8151362,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":112936,"num_write":72,"num_write_kb":19965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5762564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,12],"acting":[10,14,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[12,13,14],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"46'35","reported_seq":118,"reported_epoch":47,"state":"remapped+peering","last_fresh":"2024-09-13T11:32:42.532012+0000","last_change":"2024-09-13T11:32:43.541817+0000","last_active":"2024-09-13T11:32:42.532012+0000","last_peered":"2024-09-13T11:32:42.060897+0000","last_clean":"2024-09-13T11:32:42.060897+0000","last_became_active":"2024-09-13T11:32:20.827056+0000","last_became_peered":"2024-09-13T11:32:20.827056+0000","last_unstale":"2024-09-13T11:32:42.532012+0000","last_undegraded":"2024-09-13T11:32:42.532012+0000","last_fullsized":"2024-09-13T11:32:42.532012+0000","mapping_epoch":48,"log_start":"43'32","ondisk_log_start":"43'32","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":32,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8571471,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":67404,"num_write":31,"num_write_kb":10217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,13],"acting":[15,14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,13,14],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.d","version":"51'36","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.273555+0000","last_change":"2024-09-13T11:32:20.430625+0000","last_active":"2024-09-13T11:32:47.273555+0000","last_peered":"2024-09-13T11:32:47.273555+0000","last_clean":"2024-09-13T11:32:47.273555+0000","last_became_active":"2024-09-13T11:32:20.430258+0000","last_became_peered":"2024-09-13T11:32:20.430258+0000","last_unstale":"2024-09-13T11:32:47.273555+0000","last_undegraded":"2024-09-13T11:32:47.273555+0000","last_fullsized":"2024-09-13T11:32:47.273555+0000","mapping_epoch":31,"log_start":"51'33","ondisk_log_start":"51'33","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":33,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:48.176272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916031,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":109,"num_read_kb":81821,"num_write":34,"num_write_kb":9045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"45'16","reported_seq":78,"reported_epoch":47,"state":"peering","last_fresh":"2024-09-13T11:32:42.531575+0000","last_change":"2024-09-13T11:32:43.541732+0000","last_active":"2024-09-13T11:32:43.541732+0000","last_peered":"2024-09-13T11:32:42.531575+0000","last_clean":"2024-09-13T11:32:42.531575+0000","last_became_active":"2024-09-13T11:32:20.428069+0000","last_became_peered":"2024-09-13T11:32:20.428069+0000","last_unstale":"2024-09-13T11:32:42.531575+0000","last_undegraded":"2024-09-13T11:32:42.531575+0000","last_fullsized":"2024-09-13T11:32:42.531575+0000","mapping_epoch":48,"log_start":"43'13","ondisk_log_start":"43'13","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":13,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865577,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":21562,"num_write":14,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,12,7],"acting":[15,12,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7,9,12],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.b","version":"51'28","reported_seq":111,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-13T11:32:47.273533+0000","last_change":"2024-09-13T11:32:47.273533+0000","last_active":"2024-09-13T11:32:47.273533+0000","last_peered":"2024-09-13T11:32:46.331583+0000","last_clean":"2024-09-13T11:32:46.331583+0000","last_became_active":"2024-09-13T11:32:46.277169+0000","last_became_peered":"2024-09-13T11:32:46.277169+0000","last_unstale":"2024-09-13T11:32:47.273533+0000","last_undegraded":"2024-09-13T11:32:47.273533+0000","last_fullsized":"2024-09-13T11:32:47.273533+0000","mapping_epoch":52,"log_start":"44'25","ondisk_log_start":"44'25","created":31,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":25,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7596698,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":40414,"num_write":26,"num_write_kb":7597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4770005,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.a","version":"45'20","reported_seq":125,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.285716+0000","last_change":"2024-09-13T11:32:47.284684+0000","last_active":"2024-09-13T11:32:47.285716+0000","last_peered":"2024-09-13T11:32:47.285716+0000","last_clean":"2024-09-13T11:32:47.285716+0000","last_became_active":"2024-09-13T11:32:47.279408+0000","last_became_peered":"2024-09-13T11:32:47.279408+0000","last_unstale":"2024-09-13T11:32:47.285716+0000","last_undegraded":"2024-09-13T11:32:47.285716+0000","last_fullsized":"2024-09-13T11:32:47.285716+0000","mapping_epoch":51,"log_start":"39'17","ondisk_log_start":"39'17","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:43.583605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4806769,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":52184,"num_write":20,"num_write_kb":5013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2979858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.9","version":"51'17","reported_seq":93,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.269082+0000","last_change":"2024-09-13T11:32:20.552611+0000","last_active":"2024-09-13T11:32:47.269082+0000","last_peered":"2024-09-13T11:32:47.269082+0000","last_clean":"2024-09-13T11:32:47.269082+0000","last_became_active":"2024-09-13T11:32:20.552319+0000","last_became_peered":"2024-09-13T11:32:20.552319+0000","last_unstale":"2024-09-13T11:32:47.269082+0000","last_undegraded":"2024-09-13T11:32:47.269082+0000","last_fullsized":"2024-09-13T11:32:47.269082+0000","mapping_epoch":31,"log_start":"41'14","ondisk_log_start":"41'14","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:37.477639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420208,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":30411,"num_write":13,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14,7],"acting":[0,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"51'41","reported_seq":137,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.275186+0000","last_change":"2024-09-13T11:32:20.430909+0000","last_active":"2024-09-13T11:32:47.275186+0000","last_peered":"2024-09-13T11:32:47.275186+0000","last_clean":"2024-09-13T11:32:47.275186+0000","last_became_active":"2024-09-13T11:32:20.430293+0000","last_became_peered":"2024-09-13T11:32:20.430293+0000","last_unstale":"2024-09-13T11:32:47.275186+0000","last_undegraded":"2024-09-13T11:32:47.275186+0000","last_fullsized":"2024-09-13T11:32:47.275186+0000","mapping_epoch":31,"log_start":"45'38","ondisk_log_start":"45'38","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:34.484622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9546310,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":107963,"num_write":39,"num_write_kb":12276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"45'33","reported_seq":138,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.272841+0000","last_change":"2024-09-13T11:32:46.274620+0000","last_active":"2024-09-13T11:32:47.272841+0000","last_peered":"2024-09-13T11:32:47.272841+0000","last_clean":"2024-09-13T11:32:47.272841+0000","last_became_active":"2024-09-13T11:32:46.274163+0000","last_became_peered":"2024-09-13T11:32:46.274163+0000","last_unstale":"2024-09-13T11:32:47.272841+0000","last_undegraded":"2024-09-13T11:32:47.272841+0000","last_fullsized":"2024-09-13T11:32:47.272841+0000","mapping_epoch":50,"log_start":"44'30","ondisk_log_start":"44'30","created":31,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:40.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916031,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":50861,"num_write":32,"num_write_kb":9576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3724484,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":171,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-13T11:32:47.269352+0000","last_change":"2024-09-13T11:32:47.269352+0000","last_active":"2024-09-13T11:32:47.269352+0000","last_peered":"2024-09-13T11:32:46.287799+0000","last_clean":"2024-09-13T11:32:46.287799+0000","last_became_active":"2024-09-13T11:32:46.275955+0000","last_became_peered":"2024-09-13T11:32:46.275955+0000","last_unstale":"2024-09-13T11:32:47.269352+0000","last_undegraded":"2024-09-13T11:32:47.269352+0000","last_fullsized":"2024-09-13T11:32:47.269352+0000","mapping_epoch":52,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:32:03.312547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":25,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T11:32:43.559841+0000","last_change":"2024-09-13T11:32:43.558746+0000","last_active":"2024-09-13T11:32:43.559841+0000","last_peered":"2024-09-13T11:32:43.559841+0000","last_clean":"2024-09-13T11:32:43.559841+0000","last_became_active":"2024-09-13T11:32:43.548587+0000","last_became_peered":"2024-09-13T11:32:43.548587+0000","last_unstale":"2024-09-13T11:32:43.559841+0000","last_undegraded":"2024-09-13T11:32:43.559841+0000","last_fullsized":"2024-09-13T11:32:43.559841+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:10.424519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,8],"acting":[15,13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.2","version":"51'125","reported_seq":236,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-13T11:32:47.268213+0000","last_change":"2024-09-13T11:32:47.268213+0000","last_active":"2024-09-13T11:32:47.268213+0000","last_peered":"2024-09-13T11:32:46.425307+0000","last_clean":"2024-09-13T11:32:46.425307+0000","last_became_active":"2024-09-13T11:32:46.275759+0000","last_became_peered":"2024-09-13T11:32:46.275759+0000","last_unstale":"2024-09-13T11:32:47.268213+0000","last_undegraded":"2024-09-13T11:32:47.268213+0000","last_fullsized":"2024-09-13T11:32:47.268213+0000","mapping_epoch":52,"log_start":"48'122","ondisk_log_start":"48'122","created":30,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":137,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16059929,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":108,"num_read_kb":1511,"num_write":144,"num_write_kb":45413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":16251716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,15],"acting":[12,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.6","version":"51'53","reported_seq":173,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.267598+0000","last_change":"2024-09-13T11:32:20.429273+0000","last_active":"2024-09-13T11:32:47.267598+0000","last_peered":"2024-09-13T11:32:47.267598+0000","last_clean":"2024-09-13T11:32:47.267598+0000","last_became_active":"2024-09-13T11:32:20.428825+0000","last_became_peered":"2024-09-13T11:32:20.428825+0000","last_unstale":"2024-09-13T11:32:47.267598+0000","last_undegraded":"2024-09-13T11:32:47.267598+0000","last_fullsized":"2024-09-13T11:32:47.267598+0000","mapping_epoch":31,"log_start":"45'50","ondisk_log_start":"45'50","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:28.563898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10168136,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":146,"num_read_kb":103690,"num_write":51,"num_write_kb":16919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,10,6],"acting":[12,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":68,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.276581+0000","last_change":"2024-09-13T11:32:08.377316+0000","last_active":"2024-09-13T11:32:47.276581+0000","last_peered":"2024-09-13T11:32:47.276581+0000","last_clean":"2024-09-13T11:32:47.276581+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:32:47.276581+0000","last_undegraded":"2024-09-13T11:32:47.276581+0000","last_fullsized":"2024-09-13T11:32:47.276581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:26.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"52'151","reported_seq":304,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.288105+0000","last_change":"2024-09-13T11:32:47.277934+0000","last_active":"2024-09-13T11:32:47.288105+0000","last_peered":"2024-09-13T11:32:47.288105+0000","last_clean":"2024-09-13T11:32:47.288105+0000","last_became_active":"2024-09-13T11:32:47.277392+0000","last_became_peered":"2024-09-13T11:32:47.277392+0000","last_unstale":"2024-09-13T11:32:47.288105+0000","last_undegraded":"2024-09-13T11:32:47.288105+0000","last_fullsized":"2024-09-13T11:32:47.288105+0000","mapping_epoch":51,"log_start":"50'145","ondisk_log_start":"50'145","created":30,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":160,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:40.554823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24372169,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":130,"num_read_kb":2857,"num_write":177,"num_write_kb":52255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":28975184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,0],"acting":[6,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":62,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.273102+0000","last_change":"2024-09-13T11:32:45.278863+0000","last_active":"2024-09-13T11:32:47.273102+0000","last_peered":"2024-09-13T11:32:47.273102+0000","last_clean":"2024-09-13T11:32:47.273102+0000","last_became_active":"2024-09-13T11:32:45.278418+0000","last_became_peered":"2024-09-13T11:32:45.278418+0000","last_unstale":"2024-09-13T11:32:47.273102+0000","last_undegraded":"2024-09-13T11:32:47.273102+0000","last_fullsized":"2024-09-13T11:32:47.273102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:23.848575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.274595+0000","last_change":"2024-09-13T11:32:45.284485+0000","last_active":"2024-09-13T11:32:47.274595+0000","last_peered":"2024-09-13T11:32:47.274595+0000","last_clean":"2024-09-13T11:32:47.274595+0000","last_became_active":"2024-09-13T11:32:45.284134+0000","last_became_peered":"2024-09-13T11:32:45.284134+0000","last_unstale":"2024-09-13T11:32:47.274595+0000","last_undegraded":"2024-09-13T11:32:47.274595+0000","last_fullsized":"2024-09-13T11:32:47.274595+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:15.963394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,8],"acting":[14,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.0","version":"52'109","reported_seq":240,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.267609+0000","last_change":"2024-09-13T11:32:45.275707+0000","last_active":"2024-09-13T11:32:47.267609+0000","last_peered":"2024-09-13T11:32:47.267609+0000","last_clean":"2024-09-13T11:32:47.267609+0000","last_became_active":"2024-09-13T11:32:45.275294+0000","last_became_peered":"2024-09-13T11:32:45.275294+0000","last_unstale":"2024-09-13T11:32:47.267609+0000","last_undegraded":"2024-09-13T11:32:47.267609+0000","last_fullsized":"2024-09-13T11:32:47.267609+0000","mapping_epoch":49,"log_start":"51'106","ondisk_log_start":"51'106","created":30,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":112,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.813579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27467982,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":98,"num_read_kb":6330,"num_write":131,"num_write_kb":47332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":24388633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,6],"acting":[12,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.4","version":"51'58","reported_seq":187,"reported_epoch":52,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:32:47.273572+0000","last_change":"2024-09-13T11:32:45.300229+0000","last_active":"2024-09-13T11:32:47.273572+0000","last_peered":"2024-09-13T11:32:47.273572+0000","last_clean":"2024-09-13T11:32:47.273572+0000","last_became_active":"2024-09-13T11:32:45.284168+0000","last_became_peered":"2024-09-13T11:32:45.284168+0000","last_unstale":"2024-09-13T11:32:47.273572+0000","last_undegraded":"2024-09-13T11:32:47.273572+0000","last_fullsized":"2024-09-13T11:32:47.273572+0000","mapping_epoch":49,"log_start":"50'55","ondisk_log_start":"50'55","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":55,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:39.329890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7983292,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91467,"num_write":64,"num_write_kb":17764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8,5],"avail_no_missing":["3","5","8"],"object_location_counts":[{"shards":"3,5,8","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":81,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.277046+0000","last_change":"2024-09-13T11:32:43.556896+0000","last_active":"2024-09-13T11:32:47.277046+0000","last_peered":"2024-09-13T11:32:47.277046+0000","last_clean":"2024-09-13T11:32:47.277046+0000","last_became_active":"2024-09-13T11:32:43.554077+0000","last_became_peered":"2024-09-13T11:32:43.554077+0000","last_unstale":"2024-09-13T11:32:47.277046+0000","last_undegraded":"2024-09-13T11:32:47.277046+0000","last_fullsized":"2024-09-13T11:32:47.277046+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:14.431517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,11],"acting":[3,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"48'103","reported_seq":157,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:32:43.571620+0000","last_change":"2024-09-13T11:32:32.496965+0000","last_active":"2024-09-13T11:32:43.571620+0000","last_peered":"2024-09-13T11:32:43.571620+0000","last_clean":"2024-09-13T11:32:30.476628+0000","last_became_active":"2024-09-13T11:32:32.496965+0000","last_became_peered":"2024-09-13T11:32:32.496965+0000","last_unstale":"2024-09-13T11:32:43.571620+0000","last_undegraded":"2024-09-13T11:32:32.493715+0000","last_fullsized":"2024-09-13T11:32:32.493571+0000","mapping_epoch":41,"log_start":"48'100","ondisk_log_start":"48'100","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:34.511149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15271097,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":95,"num_read_kb":3341,"num_write":134,"num_write_kb":39373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7],"avail_no_missing":["13","7"],"object_location_counts":[{"shards":"7,13","objects":28}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.2","version":"50'64","reported_seq":187,"reported_epoch":52,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:32:47.286291+0000","last_change":"2024-09-13T11:32:47.279787+0000","last_active":"2024-09-13T11:32:47.286291+0000","last_peered":"2024-09-13T11:32:47.286291+0000","last_clean":"2024-09-13T11:32:47.286291+0000","last_became_active":"2024-09-13T11:32:47.279262+0000","last_became_peered":"2024-09-13T11:32:47.279262+0000","last_unstale":"2024-09-13T11:32:47.286291+0000","last_undegraded":"2024-09-13T11:32:47.286291+0000","last_fullsized":"2024-09-13T11:32:47.286291+0000","mapping_epoch":51,"log_start":"45'61","ondisk_log_start":"45'61","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:46.307867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14453954,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":79,"num_read_kb":38632,"num_write":61,"num_write_kb":17022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":9002558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12,9],"avail_no_missing":["3","9","12"],"object_location_counts":[{"shards":"3,9,12","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":68,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.268115+0000","last_change":"2024-09-13T11:32:08.624950+0000","last_active":"2024-09-13T11:32:47.268115+0000","last_peered":"2024-09-13T11:32:47.268115+0000","last_clean":"2024-09-13T11:32:47.268115+0000","last_became_active":"2024-09-13T11:32:06.755851+0000","last_became_peered":"2024-09-13T11:32:06.755851+0000","last_unstale":"2024-09-13T11:32:47.268115+0000","last_undegraded":"2024-09-13T11:32:47.268115+0000","last_fullsized":"2024-09-13T11:32:47.268115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.022985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"50'40","reported_seq":142,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.267565+0000","last_change":"2024-09-13T11:32:45.275689+0000","last_active":"2024-09-13T11:32:47.267565+0000","last_peered":"2024-09-13T11:32:47.267565+0000","last_clean":"2024-09-13T11:32:47.267565+0000","last_became_active":"2024-09-13T11:32:45.275183+0000","last_became_peered":"2024-09-13T11:32:45.275183+0000","last_unstale":"2024-09-13T11:32:47.267565+0000","last_undegraded":"2024-09-13T11:32:47.267565+0000","last_fullsized":"2024-09-13T11:32:47.267565+0000","mapping_epoch":49,"log_start":"50'37","ondisk_log_start":"50'37","created":31,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":37,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:23.636828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6100908,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":38394,"num_write":39,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4766365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,11,6],"acting":[12,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":73,"reported_epoch":52,"state":"active+undersized","last_fresh":"2024-09-13T11:32:47.272580+0000","last_change":"2024-09-13T11:32:32.498855+0000","last_active":"2024-09-13T11:32:47.272580+0000","last_peered":"2024-09-13T11:32:47.272580+0000","last_clean":"2024-09-13T11:32:30.479480+0000","last_became_active":"2024-09-13T11:32:32.498855+0000","last_became_peered":"2024-09-13T11:32:32.498855+0000","last_unstale":"2024-09-13T11:32:47.272580+0000","last_undegraded":"2024-09-13T11:32:47.272580+0000","last_fullsized":"2024-09-13T11:32:32.497672+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:35.325284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":["8","6"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.0","version":"50'47","reported_seq":160,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.282987+0000","last_change":"2024-09-13T11:32:47.281940+0000","last_active":"2024-09-13T11:32:47.282987+0000","last_peered":"2024-09-13T11:32:47.282987+0000","last_clean":"2024-09-13T11:32:47.282987+0000","last_became_active":"2024-09-13T11:32:47.277722+0000","last_became_peered":"2024-09-13T11:32:47.277722+0000","last_unstale":"2024-09-13T11:32:47.282987+0000","last_undegraded":"2024-09-13T11:32:47.282987+0000","last_fullsized":"2024-09-13T11:32:47.282987+0000","mapping_epoch":51,"log_start":"45'44","ondisk_log_start":"45'44","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":44,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:28.575059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4605118,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":56794,"num_write":49,"num_write_kb":12526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2344589,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-13T11:32:47.276833+0000","last_change":"2024-09-13T11:32:08.625099+0000","last_active":"2024-09-13T11:32:47.276833+0000","last_peered":"2024-09-13T11:32:47.276833+0000","last_clean":"2024-09-13T11:32:47.276833+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:32:47.276833+0000","last_undegraded":"2024-09-13T11:32:47.276833+0000","last_fullsized":"2024-09-13T11:32:47.276833+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:31.093378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"45'82","reported_seq":207,"reported_epoch":48,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:32:43.628100+0000","last_change":"2024-09-13T11:32:43.627482+0000","last_active":"2024-09-13T11:32:43.628100+0000","last_peered":"2024-09-13T11:32:43.628100+0000","last_clean":"2024-09-13T11:32:43.628100+0000","last_became_active":"2024-09-13T11:32:43.551844+0000","last_became_peered":"2024-09-13T11:32:43.551844+0000","last_unstale":"2024-09-13T11:32:43.628100+0000","last_undegraded":"2024-09-13T11:32:43.628100+0000","last_fullsized":"2024-09-13T11:32:43.628100+0000","mapping_epoch":47,"log_start":"45'79","ondisk_log_start":"45'79","created":31,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:49.392761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18706059,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":125,"num_read_kb":79636,"num_write":92,"num_write_kb":26118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13142158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3,13],"acting":[15,3,1],"avail_no_missing":["15","1","3","13"],"object_location_counts":[{"shards":"1,3,15","objects":6}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T11:32:43.541230+0000","last_change":"2024-09-13T11:32:08.624990+0000","last_active":"2024-09-13T11:32:43.541230+0000","last_peered":"2024-09-13T11:32:43.541230+0000","last_clean":"2024-09-13T11:32:43.541230+0000","last_became_active":"2024-09-13T11:32:06.351947+0000","last_became_peered":"2024-09-13T11:32:06.351947+0000","last_unstale":"2024-09-13T11:32:43.541230+0000","last_undegraded":"2024-09-13T11:32:43.541230+0000","last_fullsized":"2024-09-13T11:32:43.541230+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:21.025838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,6],"acting":[15,13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":118807924,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":1278,"num_read_kb":974169,"num_write":637,"num_write_kb":183039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":46492581,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":26,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268804096,"data_stored":268357940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":46,"acting":48,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":83171177,"num_objects":103,"num_object_clones":0,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":431,"num_read_kb":14039,"num_write":586,"num_write_kb":184373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":69615533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":308641792,"data_stored":308033555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":11,"acting":11,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":23,"acting":23,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247820,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71640,"kb_used_data":44616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94300200,"statfs":{"total":96636764160,"available":96563404800,"internally_reserved":0,"allocated":45686784,"data_stored":45362915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,8,9,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45652,"kb_used_data":18628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94326188,"statfs":{"total":96636764160,"available":96590016512,"internally_reserved":0,"allocated":19075072,"data_stored":18756401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78864,"kb_used_data":47680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94292976,"statfs":{"total":96636764160,"available":96556007424,"internally_reserved":0,"allocated":48824320,"data_stored":48480897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280523,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280523,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39968,"kb_used_data":12944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331872,"statfs":{"total":96636764160,"available":96595836928,"internally_reserved":0,"allocated":13254656,"data_stored":12975856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,8,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":170144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280524,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80596,"kb_used_data":53572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94291244,"statfs":{"total":96636764160,"available":96554233856,"internally_reserved":0,"allocated":54857728,"data_stored":54493263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,4,5,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280523,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42460,"kb_used_data":15436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329380,"statfs":{"total":96636764160,"available":96593285120,"internally_reserved":0,"allocated":15806464,"data_stored":15530346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":94489280523,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90940,"kb_used_data":63916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94280900,"statfs":{"total":96636764160,"available":96543641600,"internally_reserved":0,"allocated":65449984,"data_stored":65081331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,4,5,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280523,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39376,"kb_used_data":12352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332464,"statfs":{"total":96636764160,"available":96596443136,"internally_reserved":0,"allocated":12648448,"data_stored":12368277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280523,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44644,"kb_used_data":17620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327196,"statfs":{"total":96636764160,"available":96591048704,"internally_reserved":0,"allocated":18042880,"data_stored":17742966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,6,7,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313227,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69976,"kb_used_data":42952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94301864,"statfs":{"total":96636764160,"available":96565108736,"internally_reserved":0,"allocated":43982848,"data_stored":43613671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,4,6,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97960,"kb_used_data":69124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94273880,"statfs":{"total":96636764160,"available":96536453120,"internally_reserved":0,"allocated":70782976,"data_stored":70422184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24428544,"data_stored":24372169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24444928,"data_stored":24413026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23801856,"data_stored":23781575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16093184,"data_stored":16059929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":51933184,"data_stored":51840151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15331328,"data_stored":15271097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28405760,"data_stored":28348291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24428544,"data_stored":24372169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":43597824,"data_stored":43527911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":31756288,"data_stored":31658604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":24420352,"data_stored":24388633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21200896,"data_stored":21180271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18067456,"data_stored":18030231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4225584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36745216,"data_stored":36682416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14270464,"data_stored":14251747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19509248,"data_stored":19470037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":11186176,"data_stored":11177234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3227648,"data_stored":3216966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11501568,"data_stored":11483570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8892416,"data_stored":8874292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":18198528,"data_stored":18164907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15941632,"data_stored":15904087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":27897856,"data_stored":27865086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":24473600,"data_stored":24444051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12701696,"data_stored":12681517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":20750336,"data_stored":20705944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:32:49.893 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-13T11:32:49.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-13T11:32:49.925 INFO:tasks.rados.rados.0.smithi204.stdout:978: got (2) No such file or directory 2024-09-13T11:32:49.925 INFO:tasks.rados.rados.0.smithi204.stdout:978: got expected ENOENT (src dne) 2024-09-13T11:32:49.925 INFO:tasks.rados.rados.0.smithi204.stdout:980: finishing set_chunk to oid 32 2024-09-13T11:32:49.925 INFO:tasks.rados.rados.0.smithi204.stdout:980: got expected ENOENT (src dne) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:981: got (22) Invalid argument 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:972: done (8 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:973: done (7 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:974: done (6 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:975: done (5 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:976: done (4 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:978: done (3 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:979: done (2 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:980: done (1 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:981: done (0 left) 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:982: write (excl) oid 47 current snap is 0 2024-09-13T11:32:49.926 INFO:tasks.rados.rados.0.smithi204.stdout:982: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-13T11:32:49.932 INFO:tasks.rados.rados.0.smithi204.stdout:982: writing smithi20438004-47 from 655506 to 1271908 tid 1 2024-09-13T11:32:49.939 INFO:tasks.rados.rados.0.smithi204.stdout:982: writing smithi20438004-47 from 1829449 to 2504829 tid 2 2024-09-13T11:32:49.939 INFO:tasks.rados.rados.0.smithi204.stdout:983: write oid 13 current snap is 0 2024-09-13T11:32:49.939 INFO:tasks.rados.rados.0.smithi204.stdout:982: finishing write tid 1 to smithi20438004-47 2024-09-13T11:32:49.939 INFO:tasks.rados.rados.0.smithi204.stdout:983: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-13T11:32:49.946 INFO:tasks.rados.rados.0.smithi204.stdout:983: writing smithi20438004-13 from 672313 to 1363964 tid 1 2024-09-13T11:32:49.952 INFO:tasks.rados.rados.0.smithi204.stdout:983: writing smithi20438004-13 from 1887931 to 2306969 tid 2 2024-09-13T11:32:49.952 INFO:tasks.rados.rados.0.smithi204.stdout:984: read oid 10 snap -1 2024-09-13T11:32:49.952 INFO:tasks.rados.rados.0.smithi204.stdout:984: expect (ObjNum 100 snap 0 seq_num 100) 2024-09-13T11:32:49.952 INFO:tasks.rados.rados.0.smithi204.stdout:985: set_chunk oid 31 target oid 2024-09-13T11:32:49.959 INFO:tasks.rados.rados.0.smithi204.stdout:985: set_chunk oid 31 offset: 423424 length: 566784 target oid b87ff767156b3c4b9c857abea6c5638a5d300d255c388bdfc9e3532db661652b offset: 0 snap -1 2024-09-13T11:32:49.959 INFO:tasks.rados.rados.0.smithi204.stdout:986: set_chunk oid 20 target oid 2024-09-13T11:32:49.959 INFO:tasks.rados.rados.0.smithi204.stdout:986: set_chunk oid 20 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:49.959 INFO:tasks.rados.rados.0.smithi204.stdout:987: write oid 17 current snap is 0 2024-09-13T11:32:49.960 INFO:tasks.rados.rados.0.smithi204.stdout:982: finishing write tid 2 to smithi20438004-47 2024-09-13T11:32:49.960 INFO:tasks.rados.rados.0.smithi204.stdout:987: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-13T11:32:49.965 INFO:tasks.rados.rados.0.smithi204.stdout:987: writing smithi20438004-17 from 689120 to 1172373 tid 1 2024-09-13T11:32:49.974 INFO:tasks.rados.rados.0.smithi204.stdout:987: writing smithi20438004-17 from 1946413 to 2625462 tid 2 2024-09-13T11:32:49.974 INFO:tasks.rados.rados.0.smithi204.stdout:988: read oid 3 snap -1 2024-09-13T11:32:49.974 INFO:tasks.rados.rados.0.smithi204.stdout:988: expect deleted 2024-09-13T11:32:49.974 INFO:tasks.rados.rados.0.smithi204.stdout:989: tier_promote oid 19 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:982: finishing write tid 3 to smithi20438004-47 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:990: read oid 45 snap -1 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:990: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:991: tier_evict oid 11 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:991: tier_evict oid 11 snap -1 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:982: finishing write tid 4 to smithi20438004-47 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:982: oid 47 updating version 0 to 39 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:982: oid 47 version 39 is already newer than 37 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:982: oid 47 version 39 is already newer than 38 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 39 (ObjNum 158 snap 0 seq_num 158) dirty exists 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:982: left oid 47 (ObjNum 158 snap 0 seq_num 158) 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:983: finishing write tid 1 to smithi20438004-13 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:983: finishing write tid 2 to smithi20438004-13 2024-09-13T11:32:49.975 INFO:tasks.rados.rados.0.smithi204.stdout:983: finishing write tid 3 to smithi20438004-13 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:983: finishing write tid 4 to smithi20438004-13 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:983: oid 13 updating version 0 to 46 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:983: oid 13 version 46 is already newer than 45 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:983: oid 13 updating version 46 to 47 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 47 (ObjNum 159 snap 0 seq_num 159) dirty exists 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:983: left oid 13 (ObjNum 159 snap 0 seq_num 159) 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:982: done (9 left) 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:983: done (8 left) 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:992: write (excl) oid 23 current snap is 0 2024-09-13T11:32:49.976 INFO:tasks.rados.rados.0.smithi204.stdout:992: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-13T11:32:49.980 INFO:tasks.rados.rados.0.smithi204.stdout:992: writing smithi20438004-23 from 705927 to 1264429 tid 1 2024-09-13T11:32:49.987 INFO:tasks.rados.rados.0.smithi204.stdout:992: writing smithi20438004-23 from 2004895 to 2427602 tid 2 2024-09-13T11:32:49.987 INFO:tasks.rados.rados.0.smithi204.stdout:993: read oid 8 snap -1 2024-09-13T11:32:49.987 INFO:tasks.rados.rados.0.smithi204.stdout:993: expect (ObjNum 154 snap 0 seq_num 154) 2024-09-13T11:32:49.987 INFO:tasks.rados.rados.0.smithi204.stdout:994: read oid 2 snap -1 2024-09-13T11:32:49.987 INFO:tasks.rados.rados.0.smithi204.stdout:994: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:49.987 INFO:tasks.rados.rados.0.smithi204.stdout:995: set_chunk oid 33 target oid 2024-09-13T11:32:49.988 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:49.988 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:49.989 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:49.987+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:49.995 INFO:tasks.rados.rados.0.smithi204.stdout:995: set_chunk oid 33 offset: 603648 length: 478720 target oid bd55e49554c3c61173f06fa12eb20278be8115a73c5201f92828fedf1080bc85 offset: 0 snap -1 2024-09-13T11:32:49.995 INFO:tasks.rados.rados.0.smithi204.stdout:996: read oid 44 snap -1 2024-09-13T11:32:49.995 INFO:tasks.rados.rados.0.smithi204.stdout:996: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:49.995 INFO:tasks.rados.rados.0.smithi204.stdout:997: write (excl) oid 42 current snap is 0 2024-09-13T11:32:49.995 INFO:tasks.rados.rados.0.smithi204.stdout:997: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-13T11:32:50.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:32:50.002 INFO:tasks.rados.rados.0.smithi204.stdout:997: writing smithi20438004-42 from 722734 to 1356485 tid 1 2024-09-13T11:32:50.010 INFO:tasks.rados.rados.0.smithi204.stdout:997: writing smithi20438004-42 from 2063377 to 2722734 tid 2 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout:998: read oid 50 snap -1 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout:998: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout:999: read oid 15 snap -1 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout:999: expect deleted 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout:986: finishing set_chunk to oid 20 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout:986: got expected ENOENT (src dne) 2024-09-13T11:32:50.011 INFO:tasks.rados.rados.0.smithi204.stdout:984: expect (ObjNum 100 snap 0 seq_num 100) 2024-09-13T11:32:50.019 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1050734 2024-09-13T11:32:50.090 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:50.088+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:50.098 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:50.098 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:50.107 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:50.108 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:50.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:32:50.117 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:50.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:50.132 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1625494 2024-09-13T11:32:50.134 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:50.134 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:50.152 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:50.152 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:50.169 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:50.170 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:32:50.168+0000 7f91a1d9b640 0 -- 172.21.15.38:0/3711617947 >> v2:172.21.15.204:6812/501019151 conn(0x7f91a42220a0 msgr2=0x7f91a4224490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:985: finishing set_chunk to oid 31 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 38 (ObjNum 143 snap 0 seq_num 143) dirty exists 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:987: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:984: done (15 left) 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:985: done (14 left) 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:986: done (13 left) 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:1000: write (excl) oid 46 current snap is 0 2024-09-13T11:32:50.173 INFO:tasks.rados.rados.0.smithi204.stdout:1000: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-13T11:32:50.180 INFO:tasks.rados.rados.0.smithi204.stdout:1000: writing smithi20438004-46 from 739541 to 1448541 tid 1 2024-09-13T11:32:50.187 INFO:tasks.rados.rados.0.smithi204.stdout:1000: writing smithi20438004-46 from 2121859 to 2548235 tid 2 2024-09-13T11:32:50.187 INFO:tasks.rados.rados.0.smithi204.stdout:1001: read oid 40 snap -1 2024-09-13T11:32:50.187 INFO:tasks.rados.rados.0.smithi204.stdout:1001: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-13T11:32:50.187 INFO:tasks.rados.rados.0.smithi204.stdout:1002: set_chunk oid 14 target oid 2024-09-13T11:32:50.191 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:50.189+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout:1002: set_chunk oid 14 offset: 2270720 length: 364032 target oid 0964e7dbb74c471ce8ebec74439f472cf8e8e2409ddeabdab09c101390d9b913 offset: 0 snap -1 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout:991: got (22) Invalid argument 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout:989: got (0) Success 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 55 (ObjNum 96 snap 0 seq_num 96) dirty exists 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout:987: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout:987: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:50.197 INFO:tasks.rados.rados.0.smithi204.stdout:987: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:50.198 INFO:tasks.rados.rados.0.smithi204.stdout:987: oid 17 updating version 0 to 63 2024-09-13T11:32:50.198 INFO:tasks.rados.rados.0.smithi204.stdout:987: oid 17 updating version 63 to 65 2024-09-13T11:32:50.198 INFO:tasks.rados.rados.0.smithi204.stdout:987: oid 17 version 65 is already newer than 64 2024-09-13T11:32:50.198 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 65 (ObjNum 160 snap 0 seq_num 160) dirty exists 2024-09-13T11:32:50.198 INFO:tasks.rados.rados.0.smithi204.stdout:987: left oid 17 (ObjNum 160 snap 0 seq_num 160) 2024-09-13T11:32:50.198 INFO:tasks.rados.rados.0.smithi204.stdout:990: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-13T11:32:50.227 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:50.227 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:50.227 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:50.227 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:50.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:32:50.258 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3103222 2024-09-13T11:32:50.286 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2224850 2024-09-13T11:32:50.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:32:50.292 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:50.290+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:50.307 INFO:tasks.rados.rados.0.smithi204.stdout:992: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:50.307 INFO:tasks.rados.rados.0.smithi204.stdout:992: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:50.307 INFO:tasks.rados.rados.0.smithi204.stdout:993: expect (ObjNum 154 snap 0 seq_num 154) 2024-09-13T11:32:50.307 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:50.384 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:50.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:50.385 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:50.385 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:50.385 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:50.388 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:50.388 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:50.393 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:50.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:32:50.395 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:50.391+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:995: finishing set_chunk to oid 33 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 36 (ObjNum 147 snap 0 seq_num 147) dirty exists 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:987: done (15 left) 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:988: done (14 left) 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:989: done (13 left) 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:990: done (12 left) 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:991: done (11 left) 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:993: done (10 left) 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:995: done (9 left) 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:1003: write (excl) oid 34 current snap is 0 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:992: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:50.462 INFO:tasks.rados.rados.0.smithi204.stdout:992: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:50.463 INFO:tasks.rados.rados.0.smithi204.stdout:992: oid 23 updating version 0 to 58 2024-09-13T11:32:50.463 INFO:tasks.rados.rados.0.smithi204.stdout:992: oid 23 version 58 is already newer than 56 2024-09-13T11:32:50.463 INFO:tasks.rados.rados.0.smithi204.stdout:992: oid 23 version 58 is already newer than 57 2024-09-13T11:32:50.463 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 58 (ObjNum 161 snap 0 seq_num 161) dirty exists 2024-09-13T11:32:50.463 INFO:tasks.rados.rados.0.smithi204.stdout:992: left oid 23 (ObjNum 161 snap 0 seq_num 161) 2024-09-13T11:32:50.463 INFO:tasks.rados.rados.0.smithi204.stdout:996: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:50.463 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:50.493 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:50.492+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'23)", 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:50.510 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.727356190999998, 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005604375, 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779709+0000", 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.63e-07 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779721+0000", 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.204e-05 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779728+0000", 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4150000000000002e-06 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779732+0000", 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6839999999999998e-06 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779761+0000", 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9405999999999999e-05 2024-09-13T11:32:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779773+0000", 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1398e-05 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785264+0000", 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054913089999999998 2024-09-13T11:32:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785313+0000", 2024-09-13T11:32:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8659999999999998e-05 2024-09-13T11:32:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:44 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'26, mlcod=24'23)", 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.726786103999999, 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055622880000000003, 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780279+0000", 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780365+0000", 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6462e-05 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780372+0000", 2024-09-13T11:32:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780376+0000", 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784152+0000", 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037767650000000001 2024-09-13T11:32:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784173+0000", 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0899000000000001e-05 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785814+0000", 2024-09-13T11:32:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001640561 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785841+0000", 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6628e-05 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:45 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'27, mlcod=24'23)", 2024-09-13T11:32:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.726198666999998, 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005510928, 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780866+0000", 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4999999999999998e-07 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780870+0000", 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6940000000000001e-06 2024-09-13T11:32:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780877+0000", 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0249999999999997e-06 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780883+0000", 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2029999999999997e-06 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784553+0000", 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036697689999999998 2024-09-13T11:32:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784573+0000", 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9936999999999999e-05 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786355+0000", 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017819870000000001 2024-09-13T11:32:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.786377+0000", 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1863000000000001e-05 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:91 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'65, mlcod=24'61)", 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.524754403000003, 2024-09-13T11:32:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053956129999999996, 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982310+0000", 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3700000000000001e-07 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982314+0000", 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2229999999999999e-06 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982320+0000", 2024-09-13T11:32:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6499999999999999e-06 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.982324+0000", 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6780000000000002e-06 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.984345+0000", 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020208819999999999 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.984364+0000", 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8834e-05 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.987671+0000", 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033072689999999998 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.987706+0000", 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.464e-05 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 44.769884693000002, 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'105, mlcod=24'100)", 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 44.769321089999998, 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061709499999999997, 2024-09-13T11:32:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737744+0000", 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7700000000000004e-07 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737749+0000", 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737756+0000", 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7649999999999996e-06 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737760+0000", 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6650000000000001e-06 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742613+0000", 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048528909999999998 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742629+0000", 2024-09-13T11:32:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6481e-05 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743902+0000", 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012723630000000001 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743914+0000", 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2663999999999999e-05 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:140 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'106, mlcod=25'101)", 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 44.768684706999998, 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005547064, 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738382+0000", 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5909999999999998e-06 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738386+0000", 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5240000000000001e-06 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738392+0000", 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8660000000000002e-06 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738396+0000", 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6739999999999996e-06 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742884+0000", 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004487441 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742896+0000", 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1866e-05 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743915+0000", 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010193229999999999 2024-09-13T11:32:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743927+0000", 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1779e-05 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.029959066, 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:32:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:32:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:32:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.024900508000002, 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:32:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:32:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:32:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:32:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:32:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.024476686, 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:32:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.024458641999999, 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:32:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:32:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.30821787, 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:32:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:32:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:32:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.294377081, 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:32:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.195409107, 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:32:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:32:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.590 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.b 51/50 [PushOp(4:da92e835:::smithi20438004-50:head, version: 38'19, data_included: [2616802~223548], data_size: 223548, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:da92e835:::smithi20438004-50:head@38'19, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2616802, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1945991600000001, 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055783109999999999, 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312466+0000", 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7499999999999999e-07 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312658+0000", 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019212100000000001 2024-09-13T11:32:50.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312681+0000", 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3059999999999999e-05 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312694+0000", 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2408999999999999e-05 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312765+0000", 2024-09-13T11:32:50.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1755000000000007e-05 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312775+0000", 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2879999999999998e-06 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.318043+0000", 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052689030000000001 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1751 4.9 4:9441befd:::smithi20438004-8:head [write 420208~530210,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.121496327, 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060468199999999996, 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.595 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1751 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385569+0000", 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.426e-06 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386470+0000", 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090069800000000002 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386485+0000", 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5225000000000001e-05 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386495+0000", 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386574+0000", 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8911000000000004e-05 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386698+0000", 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012372800000000001 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386913+0000", 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021553499999999999 2024-09-13T11:32:50.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.388673+0000", 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017596210000000001 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.389582+0000", 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090943100000000004 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391260+0000", 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00167782 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391570+0000", 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030968499999999997 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391615+0000", 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5151999999999997e-05 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.601 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1752 4.9 4:9441befd:::smithi20438004-8:head [write 1527054~766050] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1137001700000004, 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062506259999999996, 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.602 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1752 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393365+0000", 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0720000000000001e-06 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394108+0000", 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074250399999999995 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394120+0000", 2024-09-13T11:32:50.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1990999999999999e-05 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394125+0000", 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7270000000000004e-06 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394182+0000", 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6181000000000003e-05 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394248+0000", 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6848999999999996e-05 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394333+0000", 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4970999999999997e-05 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396066+0000", 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001732546 2024-09-13T11:32:50.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397394+0000", 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013282680000000001 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399415+0000", 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002021063 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399585+0000", 2024-09-13T11:32:50.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016999699999999999 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399615+0000", 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9456999999999999e-05 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1753 4.9 4:9441befd:::smithi20438004-8:head [setxattr _header (55) in=7b,truncate 2420208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:32:50.608 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1129230830000001, 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055775579999999998, 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1753 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:32:50.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394142+0000", 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7999999999999995e-07 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394147+0000", 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2120000000000002e-06 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394154+0000", 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6980000000000001e-06 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394160+0000", 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395493+0000", 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00133299 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395574+0000", 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0944000000000003e-05 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395686+0000", 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112049 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396457+0000", 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077122900000000001 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397581+0000", 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011240989999999999 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399655+0000", 2024-09-13T11:32:50.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002074046 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399684+0000", 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8924999999999998e-05 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399719+0000", 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4459000000000003e-05 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1754 4.9 4:9441befd:::smithi20438004-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.112892607, 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057025110000000004, 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1754 2024-09-13T11:32:50.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2300000000000001e-07 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394175+0000", 2024-09-13T11:32:50.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5189999999999999e-06 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394181+0000", 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6089999999999997e-06 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394187+0000", 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7400000000000001e-06 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395928+0000", 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017413470000000001 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395983+0000", 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399771+0000", 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003787786 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399814+0000", 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2755e-05 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399874+0000", 2024-09-13T11:32:50.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0350999999999999e-05 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1959 4.e e53/51 4:7984cf84:::smithi20438004-46:head v 53'41, mlcod=52'38)", 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.31864783800000002, 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057232109999999997, 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:50.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0599999999999999e-07 2024-09-13T11:32:50.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188422+0000", 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9409999999999998e-06 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188434+0000", 2024-09-13T11:32:50.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2002e-05 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188441+0000", 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0090000000000001e-06 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188712+0000", 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027109600000000002 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188724+0000", 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1790999999999999e-05 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194092+0000", 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053680450000000001 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194140+0000", 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7521000000000001e-05 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:50.626 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:50.626 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:50.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:32:50.630 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pg_num to 26 2024-09-13T11:32:50.630 INFO:tasks.rados.rados.0.smithi204.stdout:994: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-13T11:32:50.630 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:50.630 INFO:tasks.rados.rados.0.smithi204.stdout:997: finishing write tid 1 to smithi20438004-42 2024-09-13T11:32:50.630 INFO:tasks.rados.rados.0.smithi204.stdout:997: finishing write tid 2 to smithi20438004-42 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:1003: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:1003: writing smithi20438004-34 from 756348 to 1540597 tid 1 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:1003: writing smithi20438004-34 from 2063988 to 2750375 tid 2 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:992: done (9 left) 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:994: done (8 left) 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:996: done (7 left) 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:999: done (6 left) 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:1004: delete oid 10 current snap is 0 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:997: finishing write tid 3 to smithi20438004-42 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:997: finishing write tid 4 to smithi20438004-42 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:997: oid 42 updating version 0 to 40 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:997: oid 42 updating version 40 to 41 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:997: oid 42 version 41 is already newer than 39 2024-09-13T11:32:50.631 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 41 (ObjNum 162 snap 0 seq_num 162) dirty exists 2024-09-13T11:32:50.632 INFO:tasks.rados.rados.0.smithi204.stdout:997: left oid 42 (ObjNum 162 snap 0 seq_num 162) 2024-09-13T11:32:50.632 INFO:tasks.rados.rados.0.smithi204.stdout:998: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:50.632 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:50.632 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:50.626+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:50.680 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2408242 2024-09-13T11:32:50.696 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1381107 2024-09-13T11:32:50.716 INFO:tasks.rados.rados.0.smithi204.stdout:1000: finishing write tid 1 to smithi20438004-46 2024-09-13T11:32:50.716 INFO:tasks.rados.rados.0.smithi204.stdout:1001: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-13T11:32:50.724 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1596273 2024-09-13T11:32:50.728 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:50.726+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:50.734 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:50.765 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 514216 2024-09-13T11:32:50.776 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:50.777 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:50.777 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:50.777 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:50.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:1000: finishing write tid 2 to smithi20438004-46 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:1000: finishing write tid 3 to smithi20438004-46 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:1000: finishing write tid 4 to smithi20438004-46 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:1000: oid 46 updating version 0 to 39 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:1000: oid 46 updating version 39 to 41 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:1000: oid 46 version 41 is already newer than 40 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 41 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-09-13T11:32:50.787 INFO:tasks.rados.rados.0.smithi204.stdout:1000: left oid 46 (ObjNum 163 snap 0 seq_num 163) 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:1002: finishing set_chunk to oid 14 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 54 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:1003: finishing write tid 1 to smithi20438004-34 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:997: done (6 left) 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:998: done (5 left) 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:1000: done (4 left) 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:1001: done (3 left) 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:1002: done (2 left) 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:1004: done (1 left) 2024-09-13T11:32:50.788 INFO:tasks.rados.rados.0.smithi204.stdout:1005: set_chunk oid 21 target oid 2024-09-13T11:32:50.798 INFO:tasks.rados.rados.0.smithi204.stdout:1005: set_chunk oid 21 offset: 2522624 length: 1297408 target oid 6ef2dd45e7cee8a25a4f8093233cd7ede409aae6a89b138b707a0369a0dbacc3 offset: 0 snap -1 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1006: write oid 20 current snap is 0 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1003: finishing write tid 2 to smithi20438004-34 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1003: finishing write tid 3 to smithi20438004-34 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1003: finishing write tid 4 to smithi20438004-34 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1003: oid 34 updating version 0 to 90 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1003: oid 34 updating version 90 to 91 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1003: oid 34 updating version 91 to 92 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 92 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1003: left oid 34 (ObjNum 164 snap 0 seq_num 164) 2024-09-13T11:32:50.799 INFO:tasks.rados.rados.0.smithi204.stdout:1006: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-13T11:32:50.805 INFO:tasks.rados.rados.0.smithi204.stdout:1006: writing smithi20438004-20 from 789962 to 1324709 tid 1 2024-09-13T11:32:50.812 INFO:tasks.rados.rados.0.smithi204.stdout:1006: writing smithi20438004-20 from 1780952 to 2471008 tid 2 2024-09-13T11:32:50.813 INFO:tasks.rados.rados.0.smithi204.stdout:1003: done (2 left) 2024-09-13T11:32:50.813 INFO:tasks.rados.rados.0.smithi204.stdout:1007: write (excl) oid 29 current snap is 0 2024-09-13T11:32:50.813 INFO:tasks.rados.rados.0.smithi204.stdout:1005: finishing set_chunk to oid 21 2024-09-13T11:32:50.813 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 14 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-13T11:32:50.813 INFO:tasks.rados.rados.0.smithi204.stdout:1007: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-09-13T11:32:50.818 INFO:tasks.rados.rados.0.smithi204.stdout:1007: writing smithi20438004-29 from 406769 to 1016765 tid 1 2024-09-13T11:32:50.822 INFO:tasks.rados.rados.0.smithi204.stdout:1007: writing smithi20438004-29 from 1439434 to 1873148 tid 2 2024-09-13T11:32:50.828 INFO:tasks.rados.rados.0.smithi204.stdout:1007: writing smithi20438004-29 from 2303522 to 2806769 tid 3 2024-09-13T11:32:50.828 INFO:tasks.rados.rados.0.smithi204.stdout:1005: done (2 left) 2024-09-13T11:32:50.828 INFO:tasks.rados.rados.0.smithi204.stdout:1008: read oid 14 snap -1 2024-09-13T11:32:50.828 INFO:tasks.rados.rados.0.smithi204.stdout:1008: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-13T11:32:50.829 INFO:tasks.rados.rados.0.smithi204.stdout:1009: write (excl) oid 2 current snap is 0 2024-09-13T11:32:50.829 INFO:tasks.rados.rados.0.smithi204.stdout:1006: finishing write tid 1 to smithi20438004-20 2024-09-13T11:32:50.829 INFO:tasks.rados.rados.0.smithi204.stdout:1009: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-13T11:32:50.829 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:50.827+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:50.832 INFO:tasks.rados.rados.0.smithi204.stdout:1009: writing smithi20438004-2 from 423576 to 825174 tid 1 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1009: writing smithi20438004-2 from 1497916 to 2191641 tid 2 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1010: read oid 10 snap -1 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1010: expect deleted 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1011: write oid 46 current snap is 0 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1006: finishing write tid 2 to smithi20438004-20 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1006: finishing write tid 3 to smithi20438004-20 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1006: finishing write tid 4 to smithi20438004-20 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1006: oid 20 updating version 0 to 43 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1006: oid 20 version 43 is already newer than 42 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:1006: oid 20 updating version 43 to 44 2024-09-13T11:32:50.841 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 44 (ObjNum 166 snap 0 seq_num 166) dirty exists 2024-09-13T11:32:50.842 INFO:tasks.rados.rados.0.smithi204.stdout:1006: left oid 20 (ObjNum 166 snap 0 seq_num 166) 2024-09-13T11:32:50.842 INFO:tasks.rados.rados.0.smithi204.stdout:1007: finishing write tid 1 to smithi20438004-29 2024-09-13T11:32:50.842 INFO:tasks.rados.rados.0.smithi204.stdout:1007: finishing write tid 2 to smithi20438004-29 2024-09-13T11:32:50.842 INFO:tasks.rados.rados.0.smithi204.stdout:1011: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-13T11:32:50.845 INFO:tasks.rados.rados.0.smithi204.stdout:1011: writing smithi20438004-46 from 440383 to 917230 tid 1 2024-09-13T11:32:50.851 INFO:tasks.rados.rados.0.smithi204.stdout:1011: writing smithi20438004-46 from 1556398 to 1993781 tid 2 2024-09-13T11:32:50.856 INFO:tasks.rados.rados.0.smithi204.stdout:1011: writing smithi20438004-46 from 2474721 to 2840383 tid 3 2024-09-13T11:32:50.856 INFO:tasks.rados.rados.0.smithi204.stdout:1006: done (5 left) 2024-09-13T11:32:50.856 INFO:tasks.rados.rados.0.smithi204.stdout:1012: read oid 33 snap -1 2024-09-13T11:32:50.856 INFO:tasks.rados.rados.0.smithi204.stdout:1012: expect (ObjNum 147 snap 0 seq_num 147) 2024-09-13T11:32:50.856 INFO:tasks.rados.rados.0.smithi204.stdout:1013: set_chunk oid 4 target oid 2024-09-13T11:32:50.865 INFO:tasks.rados.rados.0.smithi204.stdout:1013: set_chunk oid 4 offset: 1430528 length: 216576 target oid 84107b35dec3bd49dc7100dbbfbae69c8e7aee4cb078460f18fcc3852b3346d1 offset: 0 snap -1 2024-09-13T11:32:50.865 INFO:tasks.rados.rados.0.smithi204.stdout:1014: set_chunk oid 45 target oid 2024-09-13T11:32:50.872 INFO:tasks.rados.rados.0.smithi204.stdout:1014: set_chunk oid 45 offset: 326144 length: 568320 target oid 31b81115b939bea2dda896acf2d6a08faaa4344484537d459c36314d96e037d7 offset: 0 snap -1 2024-09-13T11:32:50.872 INFO:tasks.rados.rados.0.smithi204.stdout:1015: read oid 38 snap -1 2024-09-13T11:32:50.872 INFO:tasks.rados.rados.0.smithi204.stdout:1015: expect (ObjNum 156 snap 0 seq_num 156) 2024-09-13T11:32:50.872 INFO:tasks.rados.rados.0.smithi204.stdout:1016: read oid 9 snap -1 2024-09-13T11:32:50.872 INFO:tasks.rados.rados.0.smithi204.stdout:1016: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1017: tier_promote oid 21 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: finishing write tid 3 to smithi20438004-29 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: finishing write tid 4 to smithi20438004-29 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: finishing write tid 5 to smithi20438004-29 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: oid 29 updating version 0 to 72 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: oid 29 version 72 is already newer than 70 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: oid 29 version 72 is already newer than 69 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: oid 29 version 72 is already newer than 71 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 72 (ObjNum 167 snap 0 seq_num 167) dirty exists 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: left oid 29 (ObjNum 167 snap 0 seq_num 167) 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1009: finishing write tid 1 to smithi20438004-2 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1007: done (10 left) 2024-09-13T11:32:50.873 INFO:tasks.rados.rados.0.smithi204.stdout:1018: set_chunk oid 50 target oid 2024-09-13T11:32:50.876 INFO:tasks.rados.rados.0.smithi204.stdout:1018: set_chunk oid 50 offset: 2187776 length: 227328 target oid 783d7d5bc6f899537cda5de1466e424b6014640bbc2834a3f849f11157ddc6e4 offset: 0 snap -1 2024-09-13T11:32:50.876 INFO:tasks.rados.rados.0.smithi204.stdout:1019: tier_flush oid 37 2024-09-13T11:32:50.876 INFO:tasks.rados.rados.0.smithi204.stdout:1019: tier_flush oid 37 snap -1 2024-09-13T11:32:50.876 INFO:tasks.rados.rados.0.smithi204.stdout:1008: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-13T11:32:50.906 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:50.906 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:50.906 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:50.906 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:50.907 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:50.916 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:32:50.929 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:50.928+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:50.959 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 848946 2024-09-13T11:32:51.023 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.499765455999999, 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.025 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:32:51.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:32:51.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:32:51.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:26 4.8 4:1cf41451:::smithi20438004-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.494234043999999, 2024-09-13T11:32:51.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083581760000000001, 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 26 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528196+0000", 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.048e-06 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528601+0000", 2024-09-13T11:32:51.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040467299999999998 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528612+0000", 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.171e-05 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528620+0000", 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1140000000000006e-06 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532030+0000", 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034098779999999999 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532128+0000", 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7910000000000003e-05 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532280+0000", 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015168700000000001 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533907+0000", 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016271 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533914+0000", 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2590000000000003e-06 2024-09-13T11:32:51.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536475+0000", 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025609259999999998 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536517+0000", 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.163e-05 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536553+0000", 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6241000000000001e-05 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:27 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.493796912000001, 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082102310000000001, 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.039 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 27 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.51e-07 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528635+0000", 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8399999999999999e-06 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528639+0000", 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:32:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528642+0000", 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.19e-06 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533286+0000", 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046437859999999996 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533371+0000", 2024-09-13T11:32:51.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4900000000000004e-05 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533475+0000", 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104285 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534168+0000", 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069284800000000003 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534400+0000", 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023182699999999999 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536780+0000", 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023795880000000002 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536809+0000", 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9156e-05 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536842+0000", 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3546e-05 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:28 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.493771942999999, 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083368629999999999, 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.045 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 28 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528659+0000", 2024-09-13T11:32:51.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528664+0000", 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.42e-06 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528675+0000", 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1447999999999999e-05 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533704+0000", 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050284769999999999 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533777+0000", 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3675000000000002e-05 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536883+0000", 2024-09-13T11:32:51.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031062820000000001 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536929+0000", 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5237000000000001e-05 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536994+0000", 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5152000000000002e-05 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.810827315000001, 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:32:51.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:32:51.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.802485117, 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:32:51.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:32:51.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.782127114000001, 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.063 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:32:51.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:32:51.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:32:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.774326354999999, 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.069 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:32:51.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:32:51.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.224819546999999, 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:32:51.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:797 4.8 4:1cf41451:::smithi20438004-7:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.211456984000002, 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086937119999999993, 2024-09-13T11:32:51.081 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 797 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810974+0000", 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7010000000000001e-06 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811111+0000", 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013712699999999999 2024-09-13T11:32:51.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811123+0000", 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1854e-05 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811131+0000", 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3979999999999999e-06 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813131+0000", 2024-09-13T11:32:51.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019997740000000002 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813239+0000", 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108378 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813421+0000", 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018227099999999999 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814991+0000", 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015690599999999999 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816465+0000", 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001474784 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819575+0000", 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031099769999999999 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819626+0000", 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0609999999999998e-05 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819666+0000", 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9777999999999998e-05 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:798 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:51.087 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.211283357999999, 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087603679999999993, 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 798 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811150+0000", 2024-09-13T11:32:51.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811157+0000", 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811163+0000", 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813998+0000", 2024-09-13T11:32:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002835382 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814069+0000", 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1099000000000005e-05 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814212+0000", 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142647 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816475+0000", 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022629799999999999 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816520+0000", 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5099999999999998e-05 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819739+0000", 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032186570000000002 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819820+0000", 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1527999999999996e-05 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819906+0000", 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5845000000000002e-05 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:799 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.211242018, 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089993910000000007, 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 799 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.094 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811188+0000", 2024-09-13T11:32:51.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811192+0000", 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.566e-06 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811202+0000", 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4790000000000002e-06 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811209+0000", 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814715+0000", 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035061150000000002 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814798+0000", 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2854000000000003e-05 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819962+0000", 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051643419999999997 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820067+0000", 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010525 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820186+0000", 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118886 2024-09-13T11:32:51.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:805 4.8 4:1d367c64:::smithi20438004-45:head [read 0~2739508,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.211164677999999, 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082479100000000007, 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.099 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 805 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811265+0000", 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5400000000000002e-07 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811266+0000", 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.939e-06 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811270+0000", 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4319999999999999e-06 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811274+0000", 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4150000000000001e-06 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818160+0000", 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068854700000000003 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818409+0000", 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024941299999999998 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819512+0000", 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011029869999999999 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.262631012, 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.103 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:51.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.109 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1430 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:36.796956+0000", 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.225473391, 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073959150000000003, 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1430 2024-09-13T11:32:51.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.796956+0000", 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.796956+0000", 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.796956+0000", 2024-09-13T11:32:51.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1100000000000001e-07 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.796958+0000", 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.48e-06 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.796966+0000", 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2239999999999998e-06 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.796973+0000", 2024-09-13T11:32:51.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8859999999999998e-06 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.798603+0000", 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00163069 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.798687+0000", 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3497000000000004e-05 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.804229+0000", 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055423929999999996 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.804279+0000", 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9762000000000002e-05 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:36.804351+0000", 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2571999999999996e-05 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1484 4.8 4:1251e6cc:::smithi20438004-13:head [sparse-read 0~2885135] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.912222354000001, 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074627620000000004, 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1484 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2800000000000002e-07 2024-09-13T11:32:51.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110209+0000", 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.855e-06 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110213+0000", 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9299999999999996e-06 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110216+0000", 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2389999999999998e-06 2024-09-13T11:32:51.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.116449+0000", 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062328000000000001 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.116522+0000", 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3187000000000002e-05 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117669+0000", 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001147323 2024-09-13T11:32:51.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1485 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~3966419,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.912163916000001, 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01527358, 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.119 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1485 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2300000000000002e-07 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110269+0000", 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110274+0000", 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2349999999999999e-06 2024-09-13T11:32:51.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110278+0000", 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6100000000000002e-06 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117718+0000", 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074403120000000001 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117821+0000", 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103501 2024-09-13T11:32:51.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.125539+0000", 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077171590000000003 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1636 4.8 4:1d367c64:::smithi20438004-45:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.021996748999999, 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091050230000000003, 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.123 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1636 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000433+0000", 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1200000000000003e-07 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000436+0000", 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000443+0000", 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8619999999999997e-06 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000447+0000", 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8800000000000001e-06 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.001976+0000", 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001529774 2024-09-13T11:32:51.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.002101+0000", 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124824 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009537+0000", 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074360850000000003 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.127 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1637 4.8 4:1d367c64:::smithi20438004-45:head [sparse-read 0~2368645] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.021974220000001, 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010163363, 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1637 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000457+0000", 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000460+0000", 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3720000000000001e-06 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000463+0000", 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.225e-06 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009568+0000", 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091047630000000001 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009626+0000", 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8142e-05 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.010618+0000", 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099181000000000009 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1749 4.8 4:1d367c64:::smithi20438004-45:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2347280539999996, 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49694346099999998, 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1749 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.132 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787703+0000", 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6840000000000001e-06 2024-09-13T11:32:51.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787708+0000", 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787721+0000", 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2405000000000001e-05 2024-09-13T11:32:51.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787728+0000", 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.391e-06 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787780+0000", 2024-09-13T11:32:51.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1897000000000001e-05 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787845+0000", 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5184000000000005e-05 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284644+0000", 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49679929099999998 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.138 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.138 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1009: finishing write tid 2 to smithi20438004-2 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1009: finishing write tid 3 to smithi20438004-2 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1009: finishing write tid 4 to smithi20438004-2 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1009: oid 2 updating version 0 to 59 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1009: oid 2 updating version 59 to 60 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1009: oid 2 updating version 60 to 61 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 61 (ObjNum 168 snap 0 seq_num 168) dirty exists 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1009: left oid 2 (ObjNum 168 snap 0 seq_num 168) 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1011: finishing write tid 1 to smithi20438004-46 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1011: finishing write tid 2 to smithi20438004-46 2024-09-13T11:32:51.139 INFO:tasks.rados.rados.0.smithi204.stdout:1012: expect (ObjNum 147 snap 0 seq_num 147) 2024-09-13T11:32:51.140 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:51.137+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:51.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: finishing write tid 3 to smithi20438004-46 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: finishing write tid 4 to smithi20438004-46 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: finishing write tid 5 to smithi20438004-46 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: oid 46 updating version 0 to 42 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: oid 46 updating version 42 to 43 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: oid 46 updating version 43 to 45 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: oid 46 version 45 is already newer than 44 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 45 (ObjNum 169 snap 0 seq_num 169) dirty exists 2024-09-13T11:32:51.183 INFO:tasks.rados.rados.0.smithi204.stdout:1011: left oid 46 (ObjNum 169 snap 0 seq_num 169) 2024-09-13T11:32:51.184 INFO:tasks.rados.rados.0.smithi204.stdout:1013: finishing set_chunk to oid 4 2024-09-13T11:32:51.184 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 1430528 length: 216576 2024-09-13T11:32:51.184 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:51.184 INFO:tasks.rados.rados.0.smithi204.stdout:1014: finishing set_chunk to oid 45 2024-09-13T11:32:51.184 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 24 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-13T11:32:51.184 INFO:tasks.rados.rados.0.smithi204.stdout:1015: expect (ObjNum 156 snap 0 seq_num 156) 2024-09-13T11:32:51.239 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:51.237+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:51.248 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:51.248 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:51.248 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:51.248 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:51.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:32:51.326 INFO:tasks.rados.rados.0.smithi204.stdout:1018: finishing set_chunk to oid 50 2024-09-13T11:32:51.326 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-13T11:32:51.326 INFO:tasks.rados.rados.0.smithi204.stdout:1017: got (0) Success 2024-09-13T11:32:51.326 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 14 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-13T11:32:51.326 INFO:tasks.rados.rados.0.smithi204.stdout:1016: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:51.337 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1304550 2024-09-13T11:32:51.340 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:51.339+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:51.405 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:51.405 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:51.405 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:51.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:32:51.441 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:51.440+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:51.522 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:51.522 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:51.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:51.522 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:51.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.522 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.0 50/49 [PushOp(4:001d90c2:::smithi20438004-28:head, version: 45'43, data_included: [618496~710897,1855488~337679], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:001d90c2:::smithi20438004-28:head@45'43, size: 2218524, copy_subset: [(0, 2218524)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2193167, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:32:51.522 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.287927+0000", 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.2322448599999998, 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002312461, 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.287927+0000", 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.287927+0000", 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.287928+0000", 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.289330+0000", 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014018979999999999 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.289347+0000", 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7686999999999999e-05 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.289358+0000", 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0516000000000001e-05 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.289429+0000", 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0858000000000005e-05 2024-09-13T11:32:51.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.289440+0000", 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0776e-05 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.290240+0000", 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080012900000000001 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.0 50/49 [PushOp(4:001d90c2:::smithi20438004-28:head, version: 45'43, data_included: [2193167~25357], data_size: 25357, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:001d90c2:::smithi20438004-28:head@45'43, size: 2218524, copy_subset: [(0, 2218524)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2218524, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2193167, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.291702+0000", 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.228469875, 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094841999999999995, 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291702+0000", 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291702+0000", 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291703+0000", 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0800000000000004e-07 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291794+0000", 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0847000000000006e-05 2024-09-13T11:32:51.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291812+0000", 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7989e-05 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291830+0000", 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8233999999999999e-05 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291901+0000", 2024-09-13T11:32:51.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1234000000000003e-05 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.291910+0000", 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5450000000000003e-06 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.292651+0000", 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074096299999999997 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.0 50/49 [PushOp(4:04aeeab9:::smithi20438004-5:head, version: 45'44, data_included: [466944~835584,1716224~212992], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:04aeeab9:::smithi20438004-5:head@45'44, size: 2151296, copy_subset: [(0, 2151296)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1929216, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.293924+0000", 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.2262477260000004, 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001729953, 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.293924+0000", 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.293924+0000", 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.293925+0000", 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9100000000000002e-07 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294848+0000", 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000922367 2024-09-13T11:32:51.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294866+0000", 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294877+0000", 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1534e-05 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294940+0000", 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.232e-05 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294948+0000", 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8170000000000005e-06 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.295654+0000", 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00070584100000000002 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_backfill(progress 4.0 e 50/50 lb 4:001d90c2:::smithi20438004-28:head)", 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.294897+0000", 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.2252753350000001, 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087514500000000002, 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294897+0000", 2024-09-13T11:32:51.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294897+0000", 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294897+0000", 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7e-07 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294905+0000", 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9300000000000003e-06 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294912+0000", 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5710000000000003e-06 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.294916+0000", 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0040000000000001e-06 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.295693+0000", 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077731700000000003 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.295710+0000", 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6974999999999999e-05 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.295772+0000", 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1977999999999995e-05 2024-09-13T11:32:51.538 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1177326159999996, 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026017369999999998, 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402441+0000", 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.508e-06 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403463+0000", 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010217970000000001 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403475+0000", 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2498e-05 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403483+0000", 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4590000000000001e-06 2024-09-13T11:32:51.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403531+0000", 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-05 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403552+0000", 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0270000000000001e-05 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.404996+0000", 2024-09-13T11:32:51.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001443915 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.405041+0000", 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5540000000000001e-05 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1119676250000001, 2024-09-13T11:32:51.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028959070000000001, 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408205+0000", 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409400+0000", 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011945110000000001 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409410+0000", 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0159999999999999e-05 2024-09-13T11:32:51.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409416+0000", 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409467+0000", 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1270999999999998e-05 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409489+0000", 2024-09-13T11:32:51.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1682999999999999e-05 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411056+0000", 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015677569999999999 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411100+0000", 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4036999999999999e-05 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1703 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'56, mlcod=44'53)", 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.409491+0000", 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1106808700000004, 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001666205, 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409491+0000", 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409491+0000", 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409492+0000", 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4400000000000001e-07 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409495+0000", 2024-09-13T11:32:51.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3000000000000002e-06 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409509+0000", 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3875e-05 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409513+0000", 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6909999999999999e-06 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.410299+0000", 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00078661400000000004 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.410315+0000", 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5933000000000001e-05 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411133+0000", 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081749800000000005 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411157+0000", 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4850000000000001e-05 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1728 4.0 e50/49 4:04aeeab9:::smithi20438004-5:head v 50'45, mlcod=0'0)", 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.0759122830000001, 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002646874, 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444261+0000", 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0920000000000001e-06 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445429+0000", 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00116847 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445444+0000", 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4487999999999999e-05 2024-09-13T11:32:51.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445454+0000", 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8349999999999999e-06 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445509+0000", 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5154000000000001e-05 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445525+0000", 2024-09-13T11:32:51.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6654e-05 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.446871+0000", 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001345534 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.446907+0000", 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5646999999999999e-05 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1729 4.0 e50/49 4:04aeeab9:::smithi20438004-5:head v 50'46, mlcod=0'0)", 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.448325+0000", 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.0718475490000001, 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071528399999999995, 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448325+0000", 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448325+0000", 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448325+0000", 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7599999999999997e-07 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448442+0000", 2024-09-13T11:32:51.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011673899999999999 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448450+0000", 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6130000000000004e-06 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448459+0000", 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6850000000000007e-06 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448492+0000", 2024-09-13T11:32:51.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3438000000000001e-05 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.448518+0000", 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5089e-05 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.449017+0000", 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00049893400000000003 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.449040+0000", 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3309999999999999e-05 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 51/50 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 25'111, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@25'111, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.279691+0000", 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2404813939999997, 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090065299999999996, 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.561 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.279691+0000", 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.279691+0000", 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.279692+0000", 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.279986+0000", 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029400299999999998 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.280010+0000", 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4641e-05 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.280033+0000", 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.226e-05 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.280126+0000", 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3183999999999994e-05 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.280135+0000", 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0010000000000007e-06 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.280591+0000", 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045656100000000002 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(1.0 51/50 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 25'112, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@25'112, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.284332+0000", 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2358399220000003, 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087792199999999995, 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.565 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284332+0000", 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284332+0000", 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284333+0000", 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7600000000000006e-07 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284616+0000", 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028295799999999998 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284637+0000", 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1024000000000001e-05 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284649+0000", 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1843e-05 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284739+0000", 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0467e-05 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284754+0000", 2024-09-13T11:32:51.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5068999999999999e-05 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285210+0000", 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000455585 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_backfill(progress 1.0 e 51/51 lb 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head)", 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.284670+0000", 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2355020579999998, 2024-09-13T11:32:51.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072162799999999998, 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284670+0000", 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284670+0000", 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284670+0000", 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.84e-07 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284678+0000", 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2019999999999998e-06 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284683+0000", 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6790000000000002e-06 2024-09-13T11:32:51.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284687+0000", 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1130000000000003e-06 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285289+0000", 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060150600000000002 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285305+0000", 2024-09-13T11:32:51.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6538e-05 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285392+0000", 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6205999999999994e-05 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.3.0:40 4.4 e51/49 4:2b094039:::smithi20438004-40:head v 51'57, mlcod=50'55)", 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.285145+0000", 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2350275359999996, 2024-09-13T11:32:51.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001210336, 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285145+0000", 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285145+0000", 2024-09-13T11:32:51.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285145+0000", 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6899999999999999e-07 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285154+0000", 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3680000000000008e-06 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285165+0000", 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0890000000000001e-05 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285171+0000", 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9220000000000002e-06 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285233+0000", 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1634e-05 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.285271+0000", 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8114999999999997e-05 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.286295+0000", 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001024507 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.286355+0000", 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9731000000000001e-05 2024-09-13T11:32:51.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1779 4.4 e51/49 4:2451796c:::smithi20438004-17:head v 51'58, mlcod=50'56)", 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:47.198057+0000", 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.3221153460000004, 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001048062, 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198057+0000", 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198057+0000", 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198058+0000", 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2619999999999999e-06 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198068+0000", 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7510000000000003e-06 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198087+0000", 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8955000000000001e-05 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198095+0000", 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2349999999999999e-06 2024-09-13T11:32:51.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198177+0000", 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2262999999999995e-05 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.198212+0000", 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5043999999999999e-05 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.199074+0000", 2024-09-13T11:32:51.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00086154099999999996 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.199105+0000", 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1010999999999998e-05 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1904 4.5 e53/50 4:a5f8f9e2:::smithi20438004-47:head v 53'37, mlcod=52'35)", 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.5872304860000002, 2024-09-13T11:32:51.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025939650000000002, 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.66e-07 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933607+0000", 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00066545899999999997 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933617+0000", 2024-09-13T11:32:51.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0273e-05 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933625+0000", 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5079999999999999e-06 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933679+0000", 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4447999999999998e-05 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933701+0000", 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1342999999999999e-05 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.935495+0000", 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001794604 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.935536+0000", 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0163999999999997e-05 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1905 4.5 e53/50 4:a5f8f9e2:::smithi20438004-47:head v 53'38, mlcod=52'36)", 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.939428+0000", 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.5807439089999999, 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021342090000000002, 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.939428+0000", 2024-09-13T11:32:51.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.939428+0000", 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.939429+0000", 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3900000000000004e-07 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940004+0000", 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00057493900000000001 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940015+0000", 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1272e-05 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940021+0000", 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8769999999999998e-06 2024-09-13T11:32:51.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940062+0000", 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0998000000000003e-05 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940075+0000", 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2533e-05 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.941529+0000", 2024-09-13T11:32:51.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001454115 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.941562+0000", 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3636000000000001e-05 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1906 4.5 e53/50 4:a5f8f9e2:::smithi20438004-47:head v 53'39, mlcod=52'36)", 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.940487+0000", 2024-09-13T11:32:51.591 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.579685241, 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011199719999999999, 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940487+0000", 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940487+0000", 2024-09-13T11:32:51.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940488+0000", 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-06 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940495+0000", 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9249999999999998e-06 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940516+0000", 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1321000000000001e-05 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940524+0000", 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.43e-06 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940840+0000", 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031597899999999998 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.940855+0000", 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4929000000000001e-05 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.941580+0000", 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072574800000000002 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.941607+0000", 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6486999999999999e-05 2024-09-13T11:32:51.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1919 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'63, mlcod=53'61)", 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.966073+0000", 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.55409905, 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023158390000000001, 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966073+0000", 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966073+0000", 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966074+0000", 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7199999999999997e-07 2024-09-13T11:32:51.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966841+0000", 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076656800000000002 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966854+0000", 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3584000000000001e-05 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966867+0000", 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2924e-05 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966929+0000", 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1847000000000006e-05 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.966948+0000", 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8828999999999999e-05 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.968353+0000", 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001405407 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.968389+0000", 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5707999999999998e-05 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.600 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.5459863729999999, 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026856010000000001, 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3300000000000002e-07 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974977+0000", 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079094699999999996 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974994+0000", 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7085e-05 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975004+0000", 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3149999999999998e-06 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975070+0000", 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6167999999999994e-05 2024-09-13T11:32:51.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975097+0000", 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6855e-05 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976824+0000", 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017271039999999999 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976871+0000", 2024-09-13T11:32:51.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7494000000000003e-05 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1921 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'65, mlcod=53'62)", 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.975830+0000", 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.544342184, 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011194200000000001, 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975830+0000", 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975830+0000", 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975831+0000", 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:32:51.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975847+0000", 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5801e-05 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975857+0000", 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0509000000000001e-05 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975864+0000", 2024-09-13T11:32:51.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0419999999999997e-06 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976227+0000", 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036249999999999998 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976242+0000", 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5800000000000001e-05 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976904+0000", 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00066165899999999999 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976949+0000", 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5287000000000002e-05 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.609 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:51.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:32:51.613 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:32:51.614 INFO:tasks.rados.rados.0.smithi204.stdout:1019: got (0) Success 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 0 (ObjNum 150 snap 0 seq_num 150) dirty exists 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1008: done (11 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1009: done (10 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1010: done (9 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1011: done (8 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1012: done (7 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1013: done (6 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1014: done (5 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1015: done (4 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1016: done (3 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1017: done (2 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1018: done (1 left) 2024-09-13T11:32:51.615 INFO:tasks.rados.rados.0.smithi204.stdout:1019: done (0 left) 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1020: read oid 35 snap -1 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1020: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1021: set_chunk oid 11 target oid 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1021: set_chunk oid 11 offset: 2928128 length: 573952 target oid 173fea0155b96c03df1acf4064b6e0e1d85e3140ddda70804966673d6c194a27 offset: 0 snap -1 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1022: write oid 43 current snap is 0 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1022: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1022: writing smithi20438004-43 from 457190 to 1009286 tid 1 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1022: writing smithi20438004-43 from 1614880 to 2195921 tid 2 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1022: writing smithi20438004-43 from 2785791 to 2857190 tid 3 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1023: set_chunk oid 16 target oid 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1023: set_chunk oid 16 offset: 80896 length: 1430016 target oid 1e16e515664065a82c53f5943cbb51265aa389c962648abfc41f4b251e1b1f14 offset: 0 snap -1 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1024: read oid 34 snap -1 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1024: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-13T11:32:51.616 INFO:tasks.rados.rados.0.smithi204.stdout:1025: write (excl) oid 1 current snap is 0 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1025: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1025: writing smithi20438004-1 from 473997 to 1101342 tid 1 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1025: writing smithi20438004-1 from 1673362 to 2114414 tid 2 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1025: writing smithi20438004-1 from 2529567 to 2873997 tid 3 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1026: tier_evict oid 36 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1026: tier_evict oid 36 snap -1 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1021: finishing set_chunk to oid 11 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 49 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-09-13T11:32:51.617 INFO:tasks.rados.rados.0.smithi204.stdout:1020: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:51.617 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:51.609+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:51.619 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3467229 2024-09-13T11:32:51.629 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:51.689 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:51.711 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:51.710+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: finishing write tid 1 to smithi20438004-43 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: finishing write tid 2 to smithi20438004-43 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: finishing write tid 3 to smithi20438004-43 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: finishing write tid 4 to smithi20438004-43 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: finishing write tid 5 to smithi20438004-43 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: oid 43 updating version 0 to 95 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: oid 43 version 95 is already newer than 94 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: oid 43 updating version 95 to 97 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: oid 43 version 97 is already newer than 96 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 97 (ObjNum 170 snap 0 seq_num 170) dirty exists 2024-09-13T11:32:51.750 INFO:tasks.rados.rados.0.smithi204.stdout:1022: left oid 43 (ObjNum 170 snap 0 seq_num 170) 2024-09-13T11:32:51.751 INFO:tasks.rados.rados.0.smithi204.stdout:1020: done (6 left) 2024-09-13T11:32:51.751 INFO:tasks.rados.rados.0.smithi204.stdout:1021: done (5 left) 2024-09-13T11:32:51.751 INFO:tasks.rados.rados.0.smithi204.stdout:1022: done (4 left) 2024-09-13T11:32:51.751 INFO:tasks.rados.rados.0.smithi204.stdout:1027: write (excl) oid 38 current snap is 0 2024-09-13T11:32:51.751 INFO:tasks.rados.rados.0.smithi204.stdout:1027: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-13T11:32:51.756 INFO:tasks.rados.rados.0.smithi204.stdout:1027: writing smithi20438004-38 from 490804 to 1193398 tid 1 2024-09-13T11:32:51.759 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.759 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2042 4.2 e54/51 4:4297cb20:::smithi20438004-38:head v 54'74, mlcod=54'72)", 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.756546+0000", 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0023709899999999999, 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023874920000000002, 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.756546+0000", 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.756546+0000", 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.756547+0000", 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.35e-06 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.757553+0000", 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010062719999999999 2024-09-13T11:32:51.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.757568+0000", 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4693e-05 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.757578+0000", 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6239999999999997e-06 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:51.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.757648+0000", 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0347999999999998e-05 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.757677+0000", 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9209000000000001e-05 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 1 2024-09-13T11:32:51.763 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:51.764 INFO:tasks.rados.rados.0.smithi204.stdout:1027: writing smithi20438004-38 from 1615491 to 2200201 tid 2 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:1027: writing smithi20438004-38 from 2724284 to 2890804 tid 3 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:1028: read oid 45 snap -1 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:1028: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:1029: delete oid 14 current snap is 0 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:1023: finishing set_chunk to oid 16 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 27 (ObjNum 121 snap 0 seq_num 121) dirty exists 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:1024: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-13T11:32:51.766 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:51.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-13T11:32:51.813 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:51.811+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:51.877 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:51.877 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:51.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-13T11:32:51.912 INFO:tasks.rados.rados.0.smithi204.stdout:1025: finishing write tid 1 to smithi20438004-1 2024-09-13T11:32:51.912 INFO:tasks.rados.rados.0.smithi204.stdout:1026: got (2) No such file or directory 2024-09-13T11:32:51.912 INFO:tasks.rados.rados.0.smithi204.stdout:1026: got expected ENOENT (src dne) 2024-09-13T11:32:51.912 INFO:tasks.rados.rados.0.smithi204.stdout:1025: finishing write tid 2 to smithi20438004-1 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1023: done (6 left) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1024: done (5 left) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1026: done (4 left) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1029: done (3 left) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1030: read oid 9 snap -1 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1030: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1031: read oid 4 snap -1 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1031: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1032: read oid 42 snap -1 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1032: expect (ObjNum 162 snap 0 seq_num 162) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1033: read oid 28 snap -1 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1033: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:51.913 INFO:tasks.rados.rados.0.smithi204.stdout:1034: set_chunk oid 49 target oid 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1034: set_chunk oid 49 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1035: read oid 35 snap -1 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1035: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1036: read oid 50 snap -1 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1036: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1037: read oid 10 snap -1 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1037: expect deleted 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1038: read oid 27 snap -1 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1038: expect deleted 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1039: write oid 17 current snap is 0 2024-09-13T11:32:51.914 INFO:tasks.rados.rados.0.smithi204.stdout:1039: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-13T11:32:51.918 INFO:tasks.rados.rados.0.smithi204.stdout:1039: writing smithi20438004-17 from 524418 to 977510 tid 1 2024-09-13T11:32:51.926 INFO:tasks.rados.rados.0.smithi204.stdout:1039: writing smithi20438004-17 from 1732455 to 2320834 tid 2 2024-09-13T11:32:51.930 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:51.928+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:51.930 INFO:tasks.rados.rados.0.smithi204.stdout:1039: writing smithi20438004-17 from 2779130 to 2924418 tid 3 2024-09-13T11:32:51.930 INFO:tasks.rados.rados.0.smithi204.stdout:1040: write (excl) oid 24 current snap is 0 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: finishing write tid 3 to smithi20438004-1 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: finishing write tid 4 to smithi20438004-1 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: finishing write tid 5 to smithi20438004-1 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: oid 1 updating version 0 to 46 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: oid 1 updating version 46 to 48 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: oid 1 version 48 is already newer than 47 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: oid 1 updating version 48 to 49 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 49 (ObjNum 171 snap 0 seq_num 171) dirty exists 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1025: left oid 1 (ObjNum 171 snap 0 seq_num 171) 2024-09-13T11:32:51.931 INFO:tasks.rados.rados.0.smithi204.stdout:1040: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-13T11:32:51.936 INFO:tasks.rados.rados.0.smithi204.stdout:1040: writing smithi20438004-24 from 541225 to 1185919 tid 1 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1040: writing smithi20438004-24 from 1907290 to 2355680 tid 2 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1025: done (13 left) 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1041: read oid 39 snap -1 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1041: expect (ObjNum 141 snap 0 seq_num 141) 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1042: read oid 21 snap -1 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1042: expect (ObjNum 111 snap 0 seq_num 111) 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1027: finishing write tid 1 to smithi20438004-38 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1027: finishing write tid 2 to smithi20438004-38 2024-09-13T11:32:51.944 INFO:tasks.rados.rados.0.smithi204.stdout:1043: read oid 29 snap -1 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1043: expect (ObjNum 167 snap 0 seq_num 167) 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: finishing write tid 3 to smithi20438004-38 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: finishing write tid 4 to smithi20438004-38 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: finishing write tid 5 to smithi20438004-38 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: oid 38 updating version 0 to 76 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: oid 38 updating version 76 to 77 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: oid 38 version 77 is already newer than 75 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: oid 38 version 77 is already newer than 74 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 77 (ObjNum 172 snap 0 seq_num 172) dirty exists 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1027: left oid 38 (ObjNum 172 snap 0 seq_num 172) 2024-09-13T11:32:51.945 INFO:tasks.rados.rados.0.smithi204.stdout:1028: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-13T11:32:51.999 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:51.999 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:51.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:51.999 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:51.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:51.999 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:204 3.0 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:51.999 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.911715417, 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025518434, 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 204 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084157+0000", 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084160+0000", 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.463e-06 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084171+0000", 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1206000000000001e-05 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084199+0000", 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7796e-05 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084215+0000", 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5820000000000001e-05 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084389+0000", 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174134 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084589+0000", 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020036899999999999 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.091157+0000", 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065682249999999996 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.096484+0000", 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053268400000000002 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.102639+0000", 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061551430000000001 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109591+0000", 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006951774 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109627+0000", 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6198999999999997e-05 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.109676+0000", 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8464999999999998e-05 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:205 3.0 3:03052d75:::smithi20438004-6 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.911627133, 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.046448324999999999, 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.006 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 205 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:52.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084246+0000", 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8999999999999995e-07 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084251+0000", 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9899999999999997e-06 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084262+0000", 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.023e-05 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084270+0000", 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0669999999999992e-06 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084746+0000", 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047623199999999999 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084842+0000", 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6065999999999998e-05 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.094307+0000", 2024-09-13T11:32:52.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094653670000000006 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.098690+0000", 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043829810000000002 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.107503+0000", 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088122559999999992 2024-09-13T11:32:52.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130597+0000", 2024-09-13T11:32:52.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023094078000000001 2024-09-13T11:32:52.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130642+0000", 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5463999999999997e-05 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.130694+0000", 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1703999999999998e-05 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:213 3.0 3:22c1b591:::smithi20438004-14 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.911435089000001, 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.055117410999999998, 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.015 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 213 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084438+0000", 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5200000000000002e-07 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084441+0000", 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084447+0000", 2024-09-13T11:32:52.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5430000000000001e-06 2024-09-13T11:32:52.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084451+0000", 2024-09-13T11:32:52.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5829999999999998e-06 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.084997+0000", 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054659399999999996 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.085141+0000", 2024-09-13T11:32:52.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014359400000000001 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.097495+0000", 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012354146999999999 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.099703+0000", 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022078079999999999 2024-09-13T11:32:52.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.112967+0000", 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013264582 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139474+0000", 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026506980999999999 2024-09-13T11:32:52.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139513+0000", 2024-09-13T11:32:52.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8322999999999999e-05 2024-09-13T11:32:52.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139555+0000", 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2475999999999998e-05 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:217 3.0 3:33b1e45a:::smithi20438004-18 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.023 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.887412608999998, 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.03827937, 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 217 2024-09-13T11:32:52.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108460+0000", 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108461+0000", 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.392e-06 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108467+0000", 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8429999999999998e-06 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108478+0000", 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0243e-05 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108483+0000", 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8000000000000004e-06 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108545+0000", 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1909000000000006e-05 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.108674+0000", 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012890699999999999 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.120164+0000", 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011489868 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.122425+0000", 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002260649 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.125843+0000", 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034186220000000001 2024-09-13T11:32:52.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146587+0000", 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020743276000000001 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146664+0000", 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7330999999999995e-05 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.146739+0000", 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5530000000000004e-05 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:222 3.0 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.030 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.876535593, 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033143700999999998, 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 222 2024-09-13T11:32:52.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119337+0000", 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119339+0000", 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.452e-06 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119345+0000", 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8499999999999996e-06 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119358+0000", 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-09-13T11:32:52.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119364+0000", 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3219999999999998e-06 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119411+0000", 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6186999999999997e-05 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.119552+0000", 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000141204 2024-09-13T11:32:52.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.128873+0000", 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093210610000000003 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.131258+0000", 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023855529999999999 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.136698+0000", 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005439854 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152381+0000", 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015683002000000001 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152419+0000", 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7561000000000002e-05 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152481+0000", 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2038000000000005e-05 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:225 3.0 3:10cf6cc6:::smithi20438004-26 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.862309069999998, 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026005851, 2024-09-13T11:32:52.038 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 225 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133564+0000", 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133565+0000", 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.052e-06 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133571+0000", 2024-09-13T11:32:52.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9349999999999999e-06 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133580+0000", 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8530000000000004e-06 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133586+0000", 2024-09-13T11:32:52.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9610000000000002e-06 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133632+0000", 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5427999999999999e-05 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.133735+0000", 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102829 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147872+0000", 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014137244 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152657+0000", 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047851830000000001 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159315+0000", 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066580700000000003 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159445+0000", 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130104 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159513+0000", 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7897000000000004e-05 2024-09-13T11:32:52.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.159569+0000", 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6295000000000002e-05 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:227 3.0 3:162d2d40:::smithi20438004-28 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.045 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.856107758, 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028012617, 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227 2024-09-13T11:32:52.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139765+0000", 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139766+0000", 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139770+0000", 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0149999999999997e-06 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139779+0000", 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4460000000000007e-06 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139784+0000", 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8779999999999997e-06 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139822+0000", 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7750999999999998e-05 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.139934+0000", 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112378 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.150743+0000", 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010808768999999999 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.154264+0000", 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035207160000000001 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.164397+0000", 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010133675999999999 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167661+0000", 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032640759999999999 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167728+0000", 2024-09-13T11:32:52.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7018000000000001e-05 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167778+0000", 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9197999999999999e-05 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.053 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:232 3.0 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.848341515000001, 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032659146999999999, 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.054 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 232 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147531+0000", 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147533+0000", 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147539+0000", 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8850000000000004e-06 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147555+0000", 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6065e-05 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.147568+0000", 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3236e-05 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152097+0000", 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045292199999999996 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152211+0000", 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001141 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.162843+0000", 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010631982999999999 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167140+0000", 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042969829999999999 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.174951+0000", 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078107159999999997 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180068+0000", 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051167369999999997 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180131+0000", 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3140999999999997e-05 2024-09-13T11:32:52.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180190+0000", 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9438999999999998e-05 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:234 3.0 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.847560188999999, 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036893333, 2024-09-13T11:32:52.061 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 234 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148313+0000", 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148314+0000", 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.871e-06 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148323+0000", 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5010000000000004e-06 2024-09-13T11:32:52.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148334+0000", 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1280999999999999e-05 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.148342+0000", 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2029999999999992e-06 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152513+0000", 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041709370000000004 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.152594+0000", 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0315999999999995e-05 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165507+0000", 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012913051 2024-09-13T11:32:52.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.168150+0000", 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026431200000000001 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179750+0000", 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01160015 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185129+0000", 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053793729999999998 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185171+0000", 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2073000000000002e-05 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4456999999999999e-05 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:233 3.2 e38/30 3:53d0adee:::smithi20438004-34 low_tier:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.842343819, 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018647573000000001, 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153529+0000", 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.153530+0000", 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4920000000000001e-06 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169595+0000", 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016064407999999999 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169603+0000", 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7250000000000003e-06 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169609+0000", 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3900000000000001e-06 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169669+0000", 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0052e-05 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.169684+0000", 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4778e-05 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.172138+0000", 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024544110000000001 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.172176+0000", 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8316999999999999e-05 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.073 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:239 3.0 3:3c12287e:::smithi20438004-40 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.835793055, 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037773656000000003, 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.074 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 239 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:52.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160080+0000", 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160086+0000", 2024-09-13T11:32:52.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6810000000000003e-06 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160097+0000", 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.101e-05 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160103+0000", 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6319999999999996e-06 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160148+0000", 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.511e-05 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.160285+0000", 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000137508 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.175532+0000", 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015246902999999999 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180030+0000", 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044977300000000001 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.185903+0000", 2024-09-13T11:32:52.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058733659999999997 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197739+0000", 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011835094000000001 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197801+0000", 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2226999999999999e-05 2024-09-13T11:32:52.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197853+0000", 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2537999999999997e-05 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.081 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:241 3.0 3:2c9c5010:::smithi20438004-42 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.830023549, 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047296003000000003, 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 241 2024-09-13T11:32:52.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165849+0000", 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165851+0000", 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5269999999999999e-06 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165867+0000", 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6827e-05 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165880+0000", 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.165890+0000", 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0101999999999999e-05 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167178+0000", 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00128834 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.167324+0000", 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014554300000000001 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.178258+0000", 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010933951000000001 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.180829+0000", 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025710849999999999 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.186910+0000", 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060810639999999997 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213073+0000", 2024-09-13T11:32:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.02616305 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213112+0000", 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9292000000000003e-05 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.213145+0000", 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3126000000000001e-05 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:243 3.0 3:03808b00:::smithi20438004-44 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.818912908000001, 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042573002999999998, 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.089 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 243 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176960+0000", 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176961+0000", 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6610000000000001e-06 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176970+0000", 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2779999999999993e-06 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176982+0000", 2024-09-13T11:32:52.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2711000000000001e-05 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.176990+0000", 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000003e-06 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179794+0000", 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028036379999999998 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.179911+0000", 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117095 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.195122+0000", 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015211107 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197868+0000", 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027461540000000002 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.204148+0000", 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006280435 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219448+0000", 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015299787 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3766000000000001e-05 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219533+0000", 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0880999999999997e-05 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.095 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:251 4.6 4:6d3e8486:::smithi20438004-2:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.798027437000002, 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021332106999999999, 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.096 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 251 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197845+0000", 2024-09-13T11:32:52.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197847+0000", 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197853+0000", 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0530000000000003e-06 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197864+0000", 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0961999999999999e-05 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197874+0000", 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0377000000000001e-05 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197920+0000", 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5178e-05 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.197996+0000", 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6373999999999998e-05 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218088+0000", 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020092445 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218692+0000", 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060317200000000002 2024-09-13T11:32:52.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219008+0000", 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000316328 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219097+0000", 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8993000000000007e-05 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219140+0000", 2024-09-13T11:32:52.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3167000000000002e-05 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219177+0000", 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7301000000000001e-05 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.102 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:253 4.6 4:600f3d3f:::smithi20438004-4:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.796320933000001, 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022588031000000001, 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.103 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199552+0000", 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199553+0000", 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9299999999999996e-07 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199557+0000", 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6110000000000004e-06 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199568+0000", 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0582e-05 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199574+0000", 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3500000000000002e-06 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199623+0000", 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8736999999999998e-05 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.199673+0000", 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0232999999999998e-05 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219425+0000", 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019751614000000001 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219918+0000", 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00049309600000000005 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221826+0000", 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019085479999999999 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222084+0000", 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025730000000000002 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222114+0000", 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0048000000000001e-05 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.222140+0000", 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6018999999999999e-05 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.10.22:1 3.0 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.792597094000001, 2024-09-13T11:32:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016452626000000001, 2024-09-13T11:32:52.111 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203276+0000", 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203277+0000", 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-06 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203282+0000", 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.882e-06 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203290+0000", 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203300+0000", 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0162e-05 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203350+0000", 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9487999999999998e-05 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203422+0000", 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2537e-05 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,15", 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.203577+0000", 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015495999999999999 2024-09-13T11:32:52.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.204164+0000", 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058732999999999995 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.205037+0000", 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087292200000000004 2024-09-13T11:32:52.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219676+0000", 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014638463000000001 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219704+0000", 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8396999999999999e-05 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219728+0000", 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3944e-05 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.15.23:64 3.0 3.3342ca04 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:42.060517+0000", 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9353557620000004, 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.47916420900000001, 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060517+0000", 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060517+0000", 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060519+0000", 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.939e-06 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060526+0000", 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5320000000000001e-06 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060542+0000", 2024-09-13T11:32:52.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5852e-05 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060551+0000", 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1870000000000003e-06 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060628+0000", 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6428999999999996e-05 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060635+0000", 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539649+0000", 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.47901452 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539681+0000", 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1897000000000003e-05 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.15.23:65 3.0 3.a092811c (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:42.060578+0000", 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9352946539999998, 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.479226188, 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:32:52.124 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060578+0000", 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060578+0000", 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060579+0000", 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0399999999999996e-07 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060582+0000", 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7680000000000002e-06 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060592+0000", 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3980000000000005e-06 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060599+0000", 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2339999999999997e-06 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060660+0000", 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0534e-05 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.060666+0000", 2024-09-13T11:32:52.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7859999999999999e-06 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539795+0000", 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.47912892499999998 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539804+0000", 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9390000000000003e-06 2024-09-13T11:32:52.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1690 4.3 4.db1fe8e3 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:42.368473+0000", 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.6274001929999997, 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17124435099999999, 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1690 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368473+0000", 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368473+0000", 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368474+0000", 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.595e-06 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368479+0000", 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2830000000000001e-06 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368492+0000", 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.295e-05 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368509+0000", 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6541e-05 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368565+0000", 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5868000000000001e-05 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.368573+0000", 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6929999999999996e-06 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539669+0000", 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17109563999999999 2024-09-13T11:32:52.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:42.539717+0000", 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7781000000000002e-05 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1776 4.6 4:6978b14d:::smithi20438004-23:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:52.135 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:47.181184+0000", 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8146890820000001, 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023738513999999999, 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1776 2024-09-13T11:32:52.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181184+0000", 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181184+0000", 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:52.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181186+0000", 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4360000000000001e-06 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181197+0000", 2024-09-13T11:32:52.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1317e-05 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181228+0000", 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0608999999999998e-05 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181244+0000", 2024-09-13T11:32:52.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6007000000000001e-05 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181409+0000", 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016542 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.181585+0000", 2024-09-13T11:32:52.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017600299999999999 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:47.204922+0000", 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023336722000000001 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:52.141 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:52.143 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2663485 2024-09-13T11:32:52.143 INFO:tasks.rados.rados.0.smithi204.stdout:1034: finishing set_chunk to oid 49 2024-09-13T11:32:52.144 INFO:tasks.rados.rados.0.smithi204.stdout:1034: got expected ENOENT (src dne) 2024-09-13T11:32:52.144 INFO:tasks.rados.rados.0.smithi204.stdout:1032: expect (ObjNum 162 snap 0 seq_num 162) 2024-09-13T11:32:52.144 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 322847 2024-09-13T11:32:52.145 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:32:52.145 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:52.141+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:52.163 INFO:tasks.rados.rados.0.smithi204.stdout:1036: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-13T11:32:52.244 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:52.243+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:52.260 INFO:tasks.rados.rados.0.smithi204.stdout:1033: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:52.262 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:52.262 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:52.262 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:32:52.262 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:52.271 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:32:52.326 INFO:tasks.rados.rados.0.smithi204.stdout:1031: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:32:52.353 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1906711 2024-09-13T11:32:52.355 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:52.354+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:52.356 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1446328 2024-09-13T11:32:52.368 INFO:tasks.rados.rados.0.smithi204.stdout:1035: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:52.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:52.417 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:32:52.417 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:32:52.417 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:32:52.417 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:32:52.417 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:32:52.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:32:52.466 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:52.465+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:52.471 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 167561 2024-09-13T11:32:52.524 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:52.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:52.530 INFO:tasks.rados.rados.0.smithi204.stdout:1039: finishing write tid 1 to smithi20438004-17 2024-09-13T11:32:52.530 INFO:tasks.rados.rados.0.smithi204.stdout:1030: expect (ObjNum 101 snap 0 seq_num 101) 2024-09-13T11:32:52.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:52.542 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:52.542 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:52.547 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:52.545+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:52.559 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:52.559 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:52.569 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:52.569 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:52.575 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:52.576 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:52.577 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:32:52.593 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:52.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:52.601 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 77036 2024-09-13T11:32:52.611 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:52.611 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:52.627 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:52.644 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:52.644 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:52.647 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:52.645+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:52.663 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:52.663 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:52.669 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:52.680 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:52.801 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:52.801 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:52.801 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:52.802 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:52.802 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:52.802 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:52.802 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:52.803 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:52.803 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:52.803 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:52.804 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:52.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:52.808 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:52.808 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:52.808 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:32:52.809 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:52.805+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:52.809 INFO:tasks.rados.rados.0.smithi204.stdout:1039: finishing write tid 2 to smithi20438004-17 2024-09-13T11:32:52.809 INFO:tasks.rados.rados.0.smithi204.stdout:1039: finishing write tid 3 to smithi20438004-17 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1039: finishing write tid 4 to smithi20438004-17 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1039: finishing write tid 5 to smithi20438004-17 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1039: oid 17 updating version 0 to 68 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1039: oid 17 version 68 is already newer than 66 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1039: oid 17 version 68 is already newer than 67 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1039: oid 17 updating version 68 to 69 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 69 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1039: left oid 17 (ObjNum 174 snap 0 seq_num 174) 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1040: finishing write tid 1 to smithi20438004-24 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1027: done (15 left) 2024-09-13T11:32:52.810 INFO:tasks.rados.rados.0.smithi204.stdout:1028: done (14 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1030: done (13 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1031: done (12 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1032: done (11 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1033: done (10 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1034: done (9 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1035: done (8 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1036: done (7 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1037: done (6 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1038: done (5 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1039: done (4 left) 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1044: tier_evict oid 44 2024-09-13T11:32:52.811 INFO:tasks.rados.rados.0.smithi204.stdout:1044: tier_evict oid 44 snap -1 2024-09-13T11:32:52.812 INFO:tasks.rados.rados.0.smithi204.stdout:1042: expect (ObjNum 111 snap 0 seq_num 111) 2024-09-13T11:32:52.906 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:52.905+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:52.911 INFO:tasks.rados.rados.0.smithi204.stdout:1041: expect (ObjNum 141 snap 0 seq_num 141) 2024-09-13T11:32:52.911 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:52.918 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:52.918 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:52.926 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:32:52.966 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1113642 2024-09-13T11:32:53.007 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1368809 2024-09-13T11:32:53.008 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:53.006+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:53.029 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.029 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.032 INFO:tasks.rados.rados.0.smithi204.stdout:1040: finishing write tid 2 to smithi20438004-24 2024-09-13T11:32:53.032 INFO:tasks.rados.rados.0.smithi204.stdout:1040: finishing write tid 3 to smithi20438004-24 2024-09-13T11:32:53.032 INFO:tasks.rados.rados.0.smithi204.stdout:1040: finishing write tid 4 to smithi20438004-24 2024-09-13T11:32:53.032 INFO:tasks.rados.rados.0.smithi204.stdout:1040: oid 24 updating version 0 to 80 2024-09-13T11:32:53.032 INFO:tasks.rados.rados.0.smithi204.stdout:1040: oid 24 updating version 80 to 81 2024-09-13T11:32:53.032 INFO:tasks.rados.rados.0.smithi204.stdout:1040: oid 24 version 81 is already newer than 79 2024-09-13T11:32:53.033 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 81 (ObjNum 175 snap 0 seq_num 175) dirty exists 2024-09-13T11:32:53.033 INFO:tasks.rados.rados.0.smithi204.stdout:1040: left oid 24 (ObjNum 175 snap 0 seq_num 175) 2024-09-13T11:32:53.033 INFO:tasks.rados.rados.0.smithi204.stdout:1043: expect (ObjNum 167 snap 0 seq_num 167) 2024-09-13T11:32:53.038 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:32:53.071 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:53.108 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:53.107+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:53.141 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.141 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1044: got (22) Invalid argument 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1040: done (4 left) 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1041: done (3 left) 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1042: done (2 left) 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1043: done (1 left) 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1044: done (0 left) 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1045: read oid 28 snap -1 2024-09-13T11:32:53.148 INFO:tasks.rados.rados.0.smithi204.stdout:1045: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:53.149 INFO:tasks.rados.rados.0.smithi204.stdout:1046: write (excl) oid 21 current snap is 0 2024-09-13T11:32:53.149 INFO:tasks.rados.rados.0.smithi204.stdout:1046: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-13T11:32:53.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:32:53.155 INFO:tasks.rados.rados.0.smithi204.stdout:1046: writing smithi20438004-21 from 558032 to 1277975 tid 1 2024-09-13T11:32:53.161 INFO:tasks.rados.rados.0.smithi204.stdout:1046: writing smithi20438004-21 from 1849419 to 2441467 tid 2 2024-09-13T11:32:53.161 INFO:tasks.rados.rados.0.smithi204.stdout:1047: set_chunk oid 30 target oid 2024-09-13T11:32:53.166 INFO:tasks.rados.rados.0.smithi204.stdout:1047: set_chunk oid 30 offset: 289792 length: 219136 target oid c2aca7dcae53db3063ce22b1a758945cefc0cda9595681f68edea0f6f0a6fd1f offset: 0 snap -1 2024-09-13T11:32:53.166 INFO:tasks.rados.rados.0.smithi204.stdout:1048: tier_evict oid 33 2024-09-13T11:32:53.166 INFO:tasks.rados.rados.0.smithi204.stdout:1048: tier_evict oid 33 snap -1 2024-09-13T11:32:53.166 INFO:tasks.rados.rados.0.smithi204.stdout:1045: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:53.178 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2170237 2024-09-13T11:32:53.202 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:53.209 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:53.208+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:1046: finishing write tid 1 to smithi20438004-21 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:1046: finishing write tid 2 to smithi20438004-21 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:1046: finishing write tid 3 to smithi20438004-21 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:1046: finishing write tid 4 to smithi20438004-21 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:1046: oid 21 updating version 0 to 20 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:1046: oid 21 updating version 20 to 21 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:1046: oid 21 updating version 21 to 22 2024-09-13T11:32:53.227 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 22 (ObjNum 176 snap 0 seq_num 176) dirty exists 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1046: left oid 21 (ObjNum 176 snap 0 seq_num 176) 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1045: done (3 left) 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1046: done (2 left) 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1049: read oid 31 snap -1 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1049: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1050: write (excl) oid 32 current snap is 0 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1048: got (22) Invalid argument 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1047: finishing set_chunk to oid 30 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 49 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-13T11:32:53.228 INFO:tasks.rados.rados.0.smithi204.stdout:1050: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-13T11:32:53.234 INFO:tasks.rados.rados.0.smithi204.stdout:1050: writing smithi20438004-32 from 574839 to 1370031 tid 1 2024-09-13T11:32:53.240 INFO:tasks.rados.rados.0.smithi204.stdout:1050: writing smithi20438004-32 from 1907901 to 2359960 tid 2 2024-09-13T11:32:53.240 INFO:tasks.rados.rados.0.smithi204.stdout:1047: done (3 left) 2024-09-13T11:32:53.240 INFO:tasks.rados.rados.0.smithi204.stdout:1048: done (2 left) 2024-09-13T11:32:53.240 INFO:tasks.rados.rados.0.smithi204.stdout:1051: set_chunk oid 29 target oid 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1051: set_chunk oid 29 offset: 1606656 length: 70144 target oid a6fe2005c399f8725f8f3801bdf09a80052324f8bd011f674e42fbe4b0b53020 offset: 0 snap -1 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1052: read oid 42 snap -1 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1052: expect (ObjNum 162 snap 0 seq_num 162) 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1053: read oid 6 snap -1 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1053: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1054: read oid 15 snap -1 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1054: expect deleted 2024-09-13T11:32:53.245 INFO:tasks.rados.rados.0.smithi204.stdout:1055: set_chunk oid 17 target oid 2024-09-13T11:32:53.256 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.257 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.257 INFO:tasks.rados.rados.0.smithi204.stdout:1055: set_chunk oid 17 offset: 730112 length: 821760 target oid f4d0fac50d70bb2c6cc51b3bbd39ed889f9f40c040a545536d4d54707295d235 offset: 0 snap -1 2024-09-13T11:32:53.257 INFO:tasks.rados.rados.0.smithi204.stdout:1056: tier_evict oid 12 2024-09-13T11:32:53.257 INFO:tasks.rados.rados.0.smithi204.stdout:1056: tier_evict oid 12 snap -1 2024-09-13T11:32:53.257 INFO:tasks.rados.rados.0.smithi204.stdout:1049: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:32:53.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:32:53.310 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:53.309+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:53.339 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1965843 2024-09-13T11:32:53.351 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:53.402 INFO:tasks.rados.rados.0.smithi204.stdout:1050: finishing write tid 1 to smithi20438004-32 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1050: finishing write tid 2 to smithi20438004-32 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1050: finishing write tid 3 to smithi20438004-32 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1050: finishing write tid 4 to smithi20438004-32 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1050: oid 32 updating version 0 to 27 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1050: oid 32 updating version 27 to 28 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1050: oid 32 version 28 is already newer than 26 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 28 (ObjNum 177 snap 0 seq_num 177) dirty exists 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1050: left oid 32 (ObjNum 177 snap 0 seq_num 177) 2024-09-13T11:32:53.403 INFO:tasks.rados.rados.0.smithi204.stdout:1052: expect (ObjNum 162 snap 0 seq_num 162) 2024-09-13T11:32:53.408 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.408 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.411 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:53.409+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:53.416 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2633612 2024-09-13T11:32:53.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:32:53.417 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:53.446 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:53.476 INFO:tasks.rados.rados.0.smithi204.stdout:1051: finishing set_chunk to oid 29 2024-09-13T11:32:53.477 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 72 (ObjNum 167 snap 0 seq_num 167) dirty exists 2024-09-13T11:32:53.477 INFO:tasks.rados.rados.0.smithi204.stdout:1053: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:53.479 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1799111 2024-09-13T11:32:53.493 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 759614 2024-09-13T11:32:53.512 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:53.511+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:53.520 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.520 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.528 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:32:53.532 INFO:tasks.rados.rados.0.smithi204.stdout:1055: finishing set_chunk to oid 17 2024-09-13T11:32:53.532 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk returned error code -95 offset: 730112 length: 821760 2024-09-13T11:32:53.532 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-09-13T11:32:53.532 INFO:tasks.rados.rados.0.smithi204.stdout:1056: got (22) Invalid argument 2024-09-13T11:32:53.532 INFO:tasks.rados.rados.0.smithi204.stdout:1049: done (7 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1050: done (6 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1051: done (5 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1052: done (4 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1053: done (3 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1054: done (2 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1055: done (1 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1056: done (0 left) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1057: read oid 41 snap -1 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1057: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1058: read oid 23 snap -1 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1058: expect (ObjNum 161 snap 0 seq_num 161) 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1059: tier_evict oid 32 2024-09-13T11:32:53.533 INFO:tasks.rados.rados.0.smithi204.stdout:1059: tier_evict oid 32 snap -1 2024-09-13T11:32:53.534 INFO:tasks.rados.rados.0.smithi204.stdout:1060: tier_promote oid 26 2024-09-13T11:32:53.534 INFO:tasks.rados.rados.0.smithi204.stdout:1061: read oid 25 snap -1 2024-09-13T11:32:53.534 INFO:tasks.rados.rados.0.smithi204.stdout:1061: expect (ObjNum 129 snap 0 seq_num 129) 2024-09-13T11:32:53.534 INFO:tasks.rados.rados.0.smithi204.stdout:1062: tier_promote oid 34 2024-09-13T11:32:53.534 INFO:tasks.rados.rados.0.smithi204.stdout:1063: write oid 40 current snap is 0 2024-09-13T11:32:53.534 INFO:tasks.rados.rados.0.smithi204.stdout:1063: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-13T11:32:53.538 INFO:tasks.rados.rados.0.smithi204.stdout:1063: writing smithi20438004-40 from 591646 to 1062087 tid 1 2024-09-13T11:32:53.545 INFO:tasks.rados.rados.0.smithi204.stdout:1063: writing smithi20438004-40 from 1566383 to 2162100 tid 2 2024-09-13T11:32:53.550 INFO:tasks.rados.rados.0.smithi204.stdout:1063: writing smithi20438004-40 from 2888822 to 2991646 tid 3 2024-09-13T11:32:53.550 INFO:tasks.rados.rados.0.smithi204.stdout:1064: delete oid 12 current snap is 0 2024-09-13T11:32:53.550 INFO:tasks.rados.rados.0.smithi204.stdout:1059: got (22) Invalid argument 2024-09-13T11:32:53.550 INFO:tasks.rados.rados.0.smithi204.stdout:1060: got (0) Success 2024-09-13T11:32:53.551 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 24 (ObjNum 155 snap 0 seq_num 155) dirty exists 2024-09-13T11:32:53.551 INFO:tasks.rados.rados.0.smithi204.stdout:1057: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:32:53.551 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:53.605 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1541184 2024-09-13T11:32:53.613 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:53.611+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:53.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:32:53.634 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.634 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.642 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:32:53.658 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2351670 2024-09-13T11:32:53.663 INFO:tasks.rados.rados.0.smithi204.stdout:1061: expect (ObjNum 129 snap 0 seq_num 129) 2024-09-13T11:32:53.713 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1254976 2024-09-13T11:32:53.714 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:53.712+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:53.745 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.745 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.746 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 912435 2024-09-13T11:32:53.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1062: got (0) Success 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 92 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1057: done (7 left) 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1059: done (6 left) 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1060: done (5 left) 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1061: done (4 left) 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1062: done (3 left) 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1064: done (2 left) 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1065: read oid 25 snap -1 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1065: expect (ObjNum 129 snap 0 seq_num 129) 2024-09-13T11:32:53.776 INFO:tasks.rados.rados.0.smithi204.stdout:1066: set_chunk oid 1 target oid 2024-09-13T11:32:53.780 INFO:tasks.rados.rados.0.smithi204.stdout:1066: set_chunk oid 1 offset: 2100736 length: 235520 target oid d8cb36f942e3c86ecc31d8f8f5adfa89dd10ed4bda8d112eb1d01f12d0d8f3dc offset: 0 snap -1 2024-09-13T11:32:53.780 INFO:tasks.rados.rados.0.smithi204.stdout:1067: write (excl) oid 26 current snap is 0 2024-09-13T11:32:53.780 INFO:tasks.rados.rados.0.smithi204.stdout:1063: finishing write tid 1 to smithi20438004-40 2024-09-13T11:32:53.781 INFO:tasks.rados.rados.0.smithi204.stdout:1058: expect (ObjNum 161 snap 0 seq_num 161) 2024-09-13T11:32:53.781 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:53.814 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:53.812+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:53.870 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.870 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:32:53.915 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:53.914+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:53.967 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1096949 2024-09-13T11:32:53.982 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:53.983 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:53.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:32:54.015 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:54.014+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:54.031 INFO:tasks.rados.rados.0.smithi204.stdout:1063: finishing write tid 2 to smithi20438004-40 2024-09-13T11:32:54.031 INFO:tasks.rados.rados.0.smithi204.stdout:1063: finishing write tid 3 to smithi20438004-40 2024-09-13T11:32:54.031 INFO:tasks.rados.rados.0.smithi204.stdout:1063: finishing write tid 4 to smithi20438004-40 2024-09-13T11:32:54.031 INFO:tasks.rados.rados.0.smithi204.stdout:1063: finishing write tid 5 to smithi20438004-40 2024-09-13T11:32:54.031 INFO:tasks.rados.rados.0.smithi204.stdout:1063: oid 40 updating version 0 to 73 2024-09-13T11:32:54.031 INFO:tasks.rados.rados.0.smithi204.stdout:1063: oid 40 version 73 is already newer than 72 2024-09-13T11:32:54.032 INFO:tasks.rados.rados.0.smithi204.stdout:1063: oid 40 version 73 is already newer than 70 2024-09-13T11:32:54.032 INFO:tasks.rados.rados.0.smithi204.stdout:1063: oid 40 version 73 is already newer than 71 2024-09-13T11:32:54.032 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 73 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-09-13T11:32:54.032 INFO:tasks.rados.rados.0.smithi204.stdout:1063: left oid 40 (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:32:54.032 INFO:tasks.rados.rados.0.smithi204.stdout:1065: expect (ObjNum 129 snap 0 seq_num 129) 2024-09-13T11:32:54.032 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:54.094 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:54.094 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:54.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:32:54.116 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:54.115+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:54.150 INFO:tasks.rados.rados.0.smithi204.stdout:1066: finishing set_chunk to oid 1 2024-09-13T11:32:54.150 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 49 (ObjNum 171 snap 0 seq_num 171) dirty exists 2024-09-13T11:32:54.150 INFO:tasks.rados.rados.0.smithi204.stdout:1067: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-13T11:32:54.156 INFO:tasks.rados.rados.0.smithi204.stdout:1067: writing smithi20438004-26 from 625260 to 1246199 tid 1 2024-09-13T11:32:54.162 INFO:tasks.rados.rados.0.smithi204.stdout:1067: writing smithi20438004-26 from 1683347 to 2282733 tid 2 2024-09-13T11:32:54.166 INFO:tasks.rados.rados.0.smithi204.stdout:1067: writing smithi20438004-26 from 2943668 to 3025260 tid 3 2024-09-13T11:32:54.166 INFO:tasks.rados.rados.0.smithi204.stdout:1058: done (4 left) 2024-09-13T11:32:54.166 INFO:tasks.rados.rados.0.smithi204.stdout:1063: done (3 left) 2024-09-13T11:32:54.166 INFO:tasks.rados.rados.0.smithi204.stdout:1065: done (2 left) 2024-09-13T11:32:54.166 INFO:tasks.rados.rados.0.smithi204.stdout:1066: done (1 left) 2024-09-13T11:32:54.167 INFO:tasks.rados.rados.0.smithi204.stdout:1068: read oid 1 snap -1 2024-09-13T11:32:54.167 INFO:tasks.rados.rados.0.smithi204.stdout:1068: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-13T11:32:54.167 INFO:tasks.rados.rados.0.smithi204.stdout:1069: set_chunk oid 25 target oid 2024-09-13T11:32:54.180 INFO:tasks.rados.rados.0.smithi204.stdout:1069: set_chunk oid 25 offset: 350720 length: 1635328 target oid feb91986132ce44460ce237f660b27625024d2086a6644b6d5299aab0376245e offset: 0 snap -1 2024-09-13T11:32:54.180 INFO:tasks.rados.rados.0.smithi204.stdout:1070: write (excl) oid 5 current snap is 0 2024-09-13T11:32:54.181 INFO:tasks.rados.rados.0.smithi204.stdout:1067: finishing write tid 1 to smithi20438004-26 2024-09-13T11:32:54.181 INFO:tasks.rados.rados.0.smithi204.stdout:1070: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-13T11:32:54.187 INFO:tasks.rados.rados.0.smithi204.stdout:1070: writing smithi20438004-5 from 642067 to 1338255 tid 1 2024-09-13T11:32:54.193 INFO:tasks.rados.rados.0.smithi204.stdout:1070: writing smithi20438004-5 from 2025476 to 2484873 tid 2 2024-09-13T11:32:54.196 INFO:tasks.rados.rados.0.smithi204.stdout:1070: writing smithi20438004-5 from 2971091 to 3042067 tid 3 2024-09-13T11:32:54.197 INFO:tasks.rados.rados.0.smithi204.stdout:1071: write oid 24 current snap is 0 2024-09-13T11:32:54.197 INFO:tasks.rados.rados.0.smithi204.stdout:1071: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-13T11:32:54.204 INFO:tasks.rados.rados.0.smithi204.stdout:1071: writing smithi20438004-24 from 658874 to 1430311 tid 1 2024-09-13T11:32:54.209 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:54.209 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1071: writing smithi20438004-24 from 2083958 to 2687013 tid 2 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1072: set_chunk oid 49 target oid 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1072: set_chunk oid 49 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1073: set_chunk oid 48 target oid 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1073: set_chunk oid 48 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1074: write oid 20 current snap is 0 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1067: finishing write tid 2 to smithi20438004-26 2024-09-13T11:32:54.211 INFO:tasks.rados.rados.0.smithi204.stdout:1074: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-13T11:32:54.216 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:54.215+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:54.217 INFO:tasks.rados.rados.0.smithi204.stdout:1074: writing smithi20438004-20 from 675681 to 1238720 tid 1 2024-09-13T11:32:54.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:32:54.225 INFO:tasks.rados.rados.0.smithi204.stdout:1074: writing smithi20438004-20 from 1858793 to 2605506 tid 2 2024-09-13T11:32:54.228 INFO:tasks.rados.rados.0.smithi204.stdout:1074: writing smithi20438004-20 from 3025937 to 3075681 tid 3 2024-09-13T11:32:54.228 INFO:tasks.rados.rados.0.smithi204.stdout:1075: tier_evict oid 21 2024-09-13T11:32:54.229 INFO:tasks.rados.rados.0.smithi204.stdout:1075: tier_evict oid 21 snap -1 2024-09-13T11:32:54.229 INFO:tasks.rados.rados.0.smithi204.stdout:1067: finishing write tid 3 to smithi20438004-26 2024-09-13T11:32:54.229 INFO:tasks.rados.rados.0.smithi204.stdout:1076: set_chunk oid 12 target oid 2024-09-13T11:32:54.229 INFO:tasks.rados.rados.0.smithi204.stdout:1076: set_chunk oid 12 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:54.229 INFO:tasks.rados.rados.0.smithi204.stdout:1077: write (excl) oid 33 current snap is 0 2024-09-13T11:32:54.229 INFO:tasks.rados.rados.0.smithi204.stdout:1077: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-13T11:32:54.235 INFO:tasks.rados.rados.0.smithi204.stdout:1077: writing smithi20438004-33 from 692488 to 1330776 tid 1 2024-09-13T11:32:54.242 INFO:tasks.rados.rados.0.smithi204.stdout:1077: writing smithi20438004-33 from 1917275 to 2523999 tid 2 2024-09-13T11:32:54.243 INFO:tasks.rados.rados.0.smithi204.stdout:1078: read oid 6 snap -1 2024-09-13T11:32:54.243 INFO:tasks.rados.rados.0.smithi204.stdout:1078: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:54.243 INFO:tasks.rados.rados.0.smithi204.stdout:1079: write oid 16 current snap is 0 2024-09-13T11:32:54.243 INFO:tasks.rados.rados.0.smithi204.stdout:1079: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-13T11:32:54.250 INFO:tasks.rados.rados.0.smithi204.stdout:1079: writing smithi20438004-16 from 709295 to 1422832 tid 1 2024-09-13T11:32:54.257 INFO:tasks.rados.rados.0.smithi204.stdout:1079: writing smithi20438004-16 from 1859404 to 2609786 tid 2 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1080: write oid 42 current snap is 0 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1067: finishing write tid 4 to smithi20438004-26 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1067: finishing write tid 5 to smithi20438004-26 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1067: oid 26 updating version 0 to 31 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1067: oid 26 updating version 31 to 32 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1067: oid 26 updating version 32 to 34 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1067: oid 26 version 34 is already newer than 33 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 34 (ObjNum 180 snap 0 seq_num 180) dirty exists 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1067: left oid 26 (ObjNum 180 snap 0 seq_num 180) 2024-09-13T11:32:54.258 INFO:tasks.rados.rados.0.smithi204.stdout:1068: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-13T11:32:54.317 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:54.316+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:54.322 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:54.322 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:54.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:32:54.355 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:54.409 INFO:tasks.rados.rados.0.smithi204.stdout:1069: finishing set_chunk to oid 25 2024-09-13T11:32:54.409 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 66 (ObjNum 129 snap 0 seq_num 129) dirty exists 2024-09-13T11:32:54.409 INFO:tasks.rados.rados.0.smithi204.stdout:1072: finishing set_chunk to oid 49 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1072: got expected ENOENT (src dne) 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1073: finishing set_chunk to oid 48 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1073: got expected ENOENT (src dne) 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1074: finishing write tid 1 to smithi20438004-20 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1075: got (22) Invalid argument 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1076: finishing set_chunk to oid 12 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1076: got expected ENOENT (src dne) 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1074: finishing write tid 2 to smithi20438004-20 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1074: finishing write tid 3 to smithi20438004-20 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1074: finishing write tid 4 to smithi20438004-20 2024-09-13T11:32:54.410 INFO:tasks.rados.rados.0.smithi204.stdout:1080: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-13T11:32:54.417 INFO:tasks.rados.rados.0.smithi204.stdout:1080: writing smithi20438004-42 from 726102 to 1514888 tid 1 2024-09-13T11:32:54.419 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:54.418+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:54.422 INFO:tasks.rados.rados.0.smithi204.stdout:1080: writing smithi20438004-42 from 1917886 to 2528279 tid 2 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1080: writing smithi20438004-42 from 3108206 to 3126102 tid 3 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1067: done (13 left) 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1068: done (12 left) 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1069: done (11 left) 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1072: done (10 left) 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1073: done (9 left) 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1075: done (8 left) 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1076: done (7 left) 2024-09-13T11:32:54.427 INFO:tasks.rados.rados.0.smithi204.stdout:1081: read oid 49 snap -1 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1081: expect deleted 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1082: tier_promote oid 10 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1074: finishing write tid 5 to smithi20438004-20 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1074: oid 20 updating version 0 to 46 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1074: oid 20 updating version 46 to 47 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1074: oid 20 updating version 47 to 48 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1074: oid 20 updating version 48 to 49 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 49 (ObjNum 183 snap 0 seq_num 183) dirty exists 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1074: left oid 20 (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1077: finishing write tid 1 to smithi20438004-33 2024-09-13T11:32:54.428 INFO:tasks.rados.rados.0.smithi204.stdout:1078: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:54.460 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1982645 2024-09-13T11:32:54.470 INFO:tasks.rados.rados.0.smithi204.stdout:1077: finishing write tid 2 to smithi20438004-33 2024-09-13T11:32:54.470 INFO:tasks.rados.rados.0.smithi204.stdout:1077: finishing write tid 3 to smithi20438004-33 2024-09-13T11:32:54.470 INFO:tasks.rados.rados.0.smithi204.stdout:1077: finishing write tid 4 to smithi20438004-33 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1077: oid 33 updating version 0 to 50 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1077: oid 33 updating version 50 to 51 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1077: oid 33 updating version 51 to 52 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 52 (ObjNum 184 snap 0 seq_num 184) dirty exists 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1077: left oid 33 (ObjNum 184 snap 0 seq_num 184) 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: finishing write tid 1 to smithi20438004-16 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: finishing write tid 2 to smithi20438004-16 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: finishing write tid 3 to smithi20438004-16 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: finishing write tid 4 to smithi20438004-16 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: oid 16 updating version 0 to 32 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: oid 16 updating version 32 to 33 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: oid 16 version 33 is already newer than 31 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 33 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-13T11:32:54.471 INFO:tasks.rados.rados.0.smithi204.stdout:1079: left oid 16 (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1074: done (8 left) 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1077: done (7 left) 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1078: done (6 left) 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1079: done (5 left) 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1083: read oid 33 snap -1 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1083: expect (ObjNum 184 snap 0 seq_num 184) 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1084: delete oid 39 current snap is 0 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1080: finishing write tid 1 to smithi20438004-42 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1080: finishing write tid 2 to smithi20438004-42 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1080: finishing write tid 3 to smithi20438004-42 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1080: finishing write tid 4 to smithi20438004-42 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1080: finishing write tid 5 to smithi20438004-42 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1080: oid 42 updating version 0 to 54 2024-09-13T11:32:54.472 INFO:tasks.rados.rados.0.smithi204.stdout:1080: oid 42 updating version 54 to 55 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:1080: oid 42 version 55 is already newer than 53 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:1080: oid 42 updating version 55 to 56 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 56 (ObjNum 186 snap 0 seq_num 186) dirty exists 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:1080: left oid 42 (ObjNum 186 snap 0 seq_num 186) 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:1080: done (6 left) 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:1081: done (5 left) 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:1084: done (4 left) 2024-09-13T11:32:54.473 INFO:tasks.rados.rados.0.smithi204.stdout:1085: write (excl) oid 48 current snap is 0 2024-09-13T11:32:54.474 INFO:tasks.rados.rados.0.smithi204.stdout:1085: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-13T11:32:54.475 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:54.475 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:54.478 INFO:tasks.rados.rados.0.smithi204.stdout:1085: writing smithi20438004-48 from 759716 to 1299000 tid 1 2024-09-13T11:32:54.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:32:54.486 INFO:tasks.rados.rados.0.smithi204.stdout:1085: writing smithi20438004-48 from 2034850 to 2648912 tid 2 2024-09-13T11:32:54.486 INFO:tasks.rados.rados.0.smithi204.stdout:1086: write (excl) oid 2 current snap is 0 2024-09-13T11:32:54.486 INFO:tasks.rados.rados.0.smithi204.stdout:1086: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-13T11:32:54.493 INFO:tasks.rados.rados.0.smithi204.stdout:1086: writing smithi20438004-2 from 776523 to 1391056 tid 1 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1086: writing smithi20438004-2 from 1976979 to 2734699 tid 2 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1087: read oid 8 snap -1 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1087: expect (ObjNum 154 snap 0 seq_num 154) 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1088: read oid 29 snap -1 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1088: expect (ObjNum 167 snap 0 seq_num 167) 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1089: read oid 28 snap -1 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1089: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1090: delete oid 36 current snap is 0 2024-09-13T11:32:54.501 INFO:tasks.rados.rados.0.smithi204.stdout:1083: expect (ObjNum 184 snap 0 seq_num 184) 2024-09-13T11:32:54.525 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:54.524+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:54.579 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:54.590 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:32:54.631 INFO:tasks.rados.rados.0.smithi204.stdout:1085: finishing write tid 1 to smithi20438004-48 2024-09-13T11:32:54.631 INFO:tasks.rados.rados.0.smithi204.stdout:1085: finishing write tid 2 to smithi20438004-48 2024-09-13T11:32:54.631 INFO:tasks.rados.rados.0.smithi204.stdout:1085: finishing write tid 3 to smithi20438004-48 2024-09-13T11:32:54.631 INFO:tasks.rados.rados.0.smithi204.stdout:1085: finishing write tid 4 to smithi20438004-48 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1085: oid 48 updating version 0 to 53 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1085: oid 48 version 53 is already newer than 52 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1085: oid 48 version 53 is already newer than 51 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 53 (ObjNum 188 snap 0 seq_num 188) dirty exists 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1085: left oid 48 (ObjNum 188 snap 0 seq_num 188) 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1083: done (9 left) 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1085: done (8 left) 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1090: done (7 left) 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1091: tier_promote oid 36 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1086: finishing write tid 1 to smithi20438004-2 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1082: got (2) No such file or directory 2024-09-13T11:32:54.632 INFO:tasks.rados.rados.0.smithi204.stdout:1082: got expected ENOENT (src dne) 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 97 (ObjNum 129 snap 0 seq_num 3594710168) clean dne 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1086: finishing write tid 2 to smithi20438004-2 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1082: done (7 left) 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1092: write oid 32 current snap is 0 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1086: finishing write tid 3 to smithi20438004-2 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1086: finishing write tid 4 to smithi20438004-2 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1086: oid 2 updating version 0 to 63 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1086: oid 2 updating version 63 to 64 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1086: oid 2 updating version 64 to 65 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 65 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1086: left oid 2 (ObjNum 189 snap 0 seq_num 189) 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:1087: expect (ObjNum 154 snap 0 seq_num 154) 2024-09-13T11:32:54.633 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:54.638 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:54.638 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:54.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:32:54.653 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:54.651+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:54.721 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:54.719+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:54.722 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:54.751 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:54.751 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:54.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:32:54.773 INFO:tasks.rados.rados.0.smithi204.stdout:1089: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-13T11:32:54.797 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 229040 2024-09-13T11:32:54.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: finishing write tid 1 to smithi20438004-5 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: finishing write tid 2 to smithi20438004-5 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: finishing write tid 3 to smithi20438004-5 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: finishing write tid 4 to smithi20438004-5 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: finishing write tid 5 to smithi20438004-5 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: oid 5 updating version 0 to 52 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: oid 5 version 52 is already newer than 50 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: oid 5 version 52 is already newer than 49 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:1070: oid 5 version 52 is already newer than 51 2024-09-13T11:32:54.844 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 52 (ObjNum 181 snap 0 seq_num 181) dirty exists 2024-09-13T11:32:54.845 INFO:tasks.rados.rados.0.smithi204.stdout:1070: left oid 5 (ObjNum 181 snap 0 seq_num 181) 2024-09-13T11:32:54.845 INFO:tasks.rados.rados.0.smithi204.stdout:1092: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-13T11:32:54.848 INFO:tasks.rados.rados.0.smithi204.stdout:1092: writing smithi20438004-32 from 410137 to 891521 tid 1 2024-09-13T11:32:54.851 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:32:54.849+0000 7ff41d133640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33170) UID: 0 2024-09-13T11:32:54.855 INFO:tasks.rados.rados.0.smithi204.stdout:1092: writing smithi20438004-32 from 1410296 to 2171685 tid 2 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1092: writing smithi20438004-32 from 2728968 to 3210137 tid 3 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1070: done (7 left) 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1086: done (6 left) 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1087: done (5 left) 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1089: done (4 left) 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1093: read oid 18 snap -1 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1093: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1094: tier_promote oid 2 2024-09-13T11:32:54.862 INFO:tasks.rados.rados.0.smithi204.stdout:1091: got (2) No such file or directory 2024-09-13T11:32:54.863 INFO:tasks.rados.rados.0.smithi204.stdout:1091: got expected ENOENT (src dne) 2024-09-13T11:32:54.863 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 47 (ObjNum 129 snap 0 seq_num 3594618000) clean dne 2024-09-13T11:32:54.863 INFO:tasks.rados.rados.0.smithi204.stdout:1091: done (5 left) 2024-09-13T11:32:54.863 INFO:tasks.rados.rados.0.smithi204.stdout:1095: set_chunk oid 6 target oid 2024-09-13T11:32:54.863 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:54.863 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:54.868 INFO:tasks.rados.rados.0.smithi204.stdout:1095: set_chunk oid 6 offset: 688128 length: 467456 target oid 079d11d18d081c63ee36967d3662508dd09e4ffd8a4907f7caeee5c565b9e736 offset: 0 snap -1 2024-09-13T11:32:54.868 INFO:tasks.rados.rados.0.smithi204.stdout:1092: finishing write tid 1 to smithi20438004-32 2024-09-13T11:32:54.868 INFO:tasks.rados.rados.0.smithi204.stdout:1096: read oid 41 snap -1 2024-09-13T11:32:54.868 INFO:tasks.rados.rados.0.smithi204.stdout:1096: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:32:54.868 INFO:tasks.rados.rados.0.smithi204.stdout:1097: write (excl) oid 23 current snap is 0 2024-09-13T11:32:54.868 INFO:tasks.rados.rados.0.smithi204.stdout:1092: finishing write tid 2 to smithi20438004-32 2024-09-13T11:32:54.868 INFO:tasks.rados.rados.0.smithi204.stdout:1097: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-13T11:32:54.873 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:32:54.873 INFO:tasks.rados.rados.0.smithi204.stdout:1097: writing smithi20438004-23 from 426944 to 983577 tid 1 2024-09-13T11:32:54.879 INFO:tasks.rados.rados.0.smithi204.stdout:1097: writing smithi20438004-23 from 1468778 to 2090178 tid 2 2024-09-13T11:32:54.889 INFO:tasks.rados.rados.0.smithi204.stdout:1097: writing smithi20438004-23 from 2872744 to 3226944 tid 3 2024-09-13T11:32:54.889 INFO:tasks.rados.rados.0.smithi204.stdout:1098: write oid 25 current snap is 0 2024-09-13T11:32:54.889 INFO:tasks.rados.rados.0.smithi204.stdout:1098: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-13T11:32:54.891 INFO:tasks.rados.rados.0.smithi204.stdout:1098: writing smithi20438004-25 from 443751 to 1075633 tid 1 2024-09-13T11:32:54.899 INFO:tasks.rados.rados.0.smithi204.stdout:1098: writing smithi20438004-25 from 1810907 to 2575965 tid 2 2024-09-13T11:32:54.903 INFO:tasks.rados.rados.0.smithi204.stdout:1098: writing smithi20438004-25 from 3067461 to 3243751 tid 3 2024-09-13T11:32:54.903 INFO:tasks.rados.rados.0.smithi204.stdout:1099: read oid 38 snap -1 2024-09-13T11:32:54.904 INFO:tasks.rados.rados.0.smithi204.stdout:1099: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-13T11:32:54.904 INFO:tasks.rados.rados.0.smithi204.stdout:1100: read oid 26 snap -1 2024-09-13T11:32:54.904 INFO:tasks.rados.rados.0.smithi204.stdout:1100: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-13T11:32:54.904 INFO:tasks.rados.rados.0.smithi204.stdout:1101: read oid 27 snap -1 2024-09-13T11:32:54.904 INFO:tasks.rados.rados.0.smithi204.stdout:1101: expect deleted 2024-09-13T11:32:54.904 INFO:tasks.rados.rados.0.smithi204.stdout:1102: set_chunk oid 30 target oid 2024-09-13T11:32:54.909 INFO:tasks.rados.rados.0.smithi204.stdout:1102: set_chunk oid 30 offset: 1073664 length: 205312 target oid 209612d9808d41dba7abc5ed6737d07189151a3e0113d6d0e6c87a53e780086e offset: 0 snap -1 2024-09-13T11:32:54.909 INFO:tasks.rados.rados.0.smithi204.stdout:1103: read oid 17 snap -1 2024-09-13T11:32:54.909 INFO:tasks.rados.rados.0.smithi204.stdout:1103: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-13T11:32:54.909 INFO:tasks.rados.rados.0.smithi204.stdout:1104: set_chunk oid 45 target oid 2024-09-13T11:32:54.923 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:54.922+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:54.924 INFO:tasks.rados.rados.0.smithi204.stdout:1104: set_chunk oid 45 offset: 311808 length: 974336 target oid 5b321d6d59fd045ea6ed015f3cbc4d14b8ae61b1ea704760d73766a6fbdaad5d offset: 0 snap -1 2024-09-13T11:32:54.924 INFO:tasks.rados.rados.0.smithi204.stdout:1105: read oid 44 snap -1 2024-09-13T11:32:54.924 INFO:tasks.rados.rados.0.smithi204.stdout:1105: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:54.924 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:54.924 INFO:tasks.rados.rados.0.smithi204.stdout:1094: got (0) Success 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 65 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1094: done (15 left) 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1106: read oid 21 snap -1 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1106: expect (ObjNum 176 snap 0 seq_num 176) 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1092: finishing write tid 3 to smithi20438004-32 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1092: finishing write tid 4 to smithi20438004-32 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1092: finishing write tid 5 to smithi20438004-32 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1092: oid 32 updating version 0 to 36 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1092: oid 32 version 36 is already newer than 35 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1092: oid 32 updating version 36 to 37 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:1092: oid 32 updating version 37 to 38 2024-09-13T11:32:54.925 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 38 (ObjNum 191 snap 0 seq_num 191) dirty exists 2024-09-13T11:32:54.926 INFO:tasks.rados.rados.0.smithi204.stdout:1092: left oid 32 (ObjNum 191 snap 0 seq_num 191) 2024-09-13T11:32:54.926 INFO:tasks.rados.rados.0.smithi204.stdout:1092: done (15 left) 2024-09-13T11:32:54.926 INFO:tasks.rados.rados.0.smithi204.stdout:1107: set_chunk oid 34 target oid 2024-09-13T11:32:54.939 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:54.939 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:54.942 INFO:tasks.rados.rados.0.smithi204.stdout:1107: set_chunk oid 34 offset: 212480 length: 2402304 target oid 7d05e8c7df5e25fbd88969765a03a57ceedcb1a9494ad09c9dc5a1d441eabe1c offset: 0 snap -1 2024-09-13T11:32:54.942 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:54.942 INFO:tasks.rados.rados.0.smithi204.stdout:1095: finishing set_chunk to oid 6 2024-09-13T11:32:54.942 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 688128 length: 467456 2024-09-13T11:32:54.942 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:54.943 INFO:tasks.rados.rados.0.smithi204.stdout:1095: done (15 left) 2024-09-13T11:32:54.943 INFO:tasks.rados.rados.0.smithi204.stdout:1108: tier_evict oid 9 2024-09-13T11:32:54.943 INFO:tasks.rados.rados.0.smithi204.stdout:1108: tier_evict oid 9 snap -1 2024-09-13T11:32:54.943 INFO:tasks.rados.rados.0.smithi204.stdout:1093: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:32:54.958 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:54.958 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:54.959 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:54.975 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:54.975 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:54.986 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:54.986 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:54.986 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:54.986 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:54.991 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:54.991 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:54.995 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:32:55.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:55.009 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:55.026 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:55.026 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 31786 2024-09-13T11:32:55.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:55.027 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:55.025+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:55.042 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:55.042 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:55.058 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:55.058 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:55.074 INFO:tasks.rados.rados.0.smithi204.stdout:1096: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:32:55.074 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:55.075 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:55.092 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:55.092 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:55.109 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:55.109 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:55.125 INFO:teuthology.orchestra.run.smithi038.stdout:osd.12: {} 2024-09-13T11:32:55.125 INFO:teuthology.orchestra.run.smithi038.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-13T11:32:55.126 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:55.124+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:55.129 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:55.130 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:55.130 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:55.130 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:55.130 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:55.138 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:32:55.142 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:55.142 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:55.158 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:55.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:55.162 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1198333 2024-09-13T11:32:55.176 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:55.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:55.179 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: finishing write tid 5 to smithi20438004-23 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: oid 23 updating version 0 to 67 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: oid 23 updating version 67 to 69 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: oid 23 version 69 is already newer than 68 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: oid 23 updating version 69 to 70 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 70 (ObjNum 192 snap 0 seq_num 192) dirty exists 2024-09-13T11:32:55.193 INFO:tasks.rados.rados.0.smithi204.stdout:1097: left oid 23 (ObjNum 192 snap 0 seq_num 192) 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: finishing write tid 1 to smithi20438004-25 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: finishing write tid 2 to smithi20438004-25 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: finishing write tid 3 to smithi20438004-25 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1093: done (15 left) 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1096: done (14 left) 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1097: done (13 left) 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1109: write (excl) oid 12 current snap is 0 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: finishing write tid 4 to smithi20438004-25 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: finishing write tid 5 to smithi20438004-25 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: oid 25 updating version 0 to 75 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: oid 25 version 75 is already newer than 74 2024-09-13T11:32:55.194 INFO:tasks.rados.rados.0.smithi204.stdout:1098: oid 25 version 75 is already newer than 73 2024-09-13T11:32:55.195 INFO:tasks.rados.rados.0.smithi204.stdout:1098: oid 25 updating version 75 to 76 2024-09-13T11:32:55.195 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 76 (ObjNum 193 snap 0 seq_num 193) dirty exists 2024-09-13T11:32:55.195 INFO:tasks.rados.rados.0.smithi204.stdout:1098: left oid 25 (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:32:55.195 INFO:tasks.rados.rados.0.smithi204.stdout:1109: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-13T11:32:55.200 INFO:tasks.rados.rados.0.smithi204.stdout:1109: writing smithi20438004-12 from 460558 to 1167689 tid 1 2024-09-13T11:32:55.207 INFO:tasks.rados.rados.0.smithi204.stdout:1109: writing smithi20438004-12 from 1869389 to 2378105 tid 2 2024-09-13T11:32:55.213 INFO:tasks.rados.rados.0.smithi204.stdout:1109: writing smithi20438004-12 from 3094884 to 3260558 tid 3 2024-09-13T11:32:55.213 INFO:tasks.rados.rados.0.smithi204.stdout:1098: done (13 left) 2024-09-13T11:32:55.213 INFO:tasks.rados.rados.0.smithi204.stdout:1101: done (12 left) 2024-09-13T11:32:55.214 INFO:tasks.rados.rados.0.smithi204.stdout:1110: set_chunk oid 39 target oid 2024-09-13T11:32:55.214 INFO:tasks.rados.rados.0.smithi204.stdout:1110: set_chunk oid 39 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:55.214 INFO:tasks.rados.rados.0.smithi204.stdout:1111: write oid 46 current snap is 0 2024-09-13T11:32:55.214 INFO:tasks.rados.rados.0.smithi204.stdout:1111: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-13T11:32:55.221 INFO:tasks.rados.rados.0.smithi204.stdout:1111: writing smithi20438004-46 from 477365 to 1259745 tid 1 2024-09-13T11:32:55.226 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:55.225+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:55.230 INFO:tasks.rados.rados.0.smithi204.stdout:1111: writing smithi20438004-46 from 1927871 to 2696598 tid 2 2024-09-13T11:32:55.234 INFO:tasks.rados.rados.0.smithi204.stdout:1111: writing smithi20438004-46 from 3122307 to 3277365 tid 3 2024-09-13T11:32:55.234 INFO:tasks.rados.rados.0.smithi204.stdout:1112: set_chunk oid 2 target oid 2024-09-13T11:32:55.258 INFO:tasks.rados.rados.0.smithi204.stdout:1112: set_chunk oid 2 offset: 504320 length: 1821184 target oid 26fb968df5b522f5c349358d3d0ff8454270b3cb0aa10e2ce9da93ba717f7a03 offset: 0 snap -1 2024-09-13T11:32:55.258 INFO:tasks.rados.rados.0.smithi204.stdout:1113: write (excl) oid 8 current snap is 0 2024-09-13T11:32:55.258 INFO:tasks.rados.rados.0.smithi204.stdout:1113: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-13T11:32:55.264 INFO:tasks.rados.rados.0.smithi204.stdout:1113: writing smithi20438004-8 from 494172 to 951801 tid 1 2024-09-13T11:32:55.266 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:55.266 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'87, mlcod=24'81)", 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 50.192921009999999, 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0126724, 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.267 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071544+0000", 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7599999999999997e-07 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071634+0000", 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.9599999999999996e-05 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071640+0000", 2024-09-13T11:32:55.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4980000000000003e-06 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.071644+0000", 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2919999999999997e-06 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073407+0000", 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001762149 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.073417+0000", 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0859999999999999e-05 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084190+0000", 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010772903 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084216+0000", 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5622e-05 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:118 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'88, mlcod=24'81)", 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 50.192400996000003, 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012784627999999999, 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.272 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072064+0000", 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4900000000000001e-07 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072067+0000", 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2260000000000001e-06 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072074+0000", 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2820000000000002e-06 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.072077+0000", 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5899999999999999e-06 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075773+0000", 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003695884 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.075786+0000", 2024-09-13T11:32:55.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2627e-05 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084816+0000", 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0090302000000000004 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084848+0000", 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2369999999999997e-05 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.277 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:122 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'92, mlcod=24'82)", 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 50.190448250000003, 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013166437, 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074016+0000", 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074017+0000", 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9500000000000003e-07 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074022+0000", 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0529999999999997e-06 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074029+0000", 2024-09-13T11:32:55.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8419999999999999e-06 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074032+0000", 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8060000000000001e-06 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081498+0000", 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074659669999999996 2024-09-13T11:32:55.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.081510+0000", 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1265e-05 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087152+0000", 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056421889999999997 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087183+0000", 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.082e-05 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4263.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'82)", 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 50.189936156999998, 2024-09-13T11:32:55.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012918696, 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074528+0000", 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074529+0000", 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8400000000000004e-07 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074601+0000", 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2106000000000004e-05 2024-09-13T11:32:55.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074611+0000", 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0054000000000001e-05 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.074616+0000", 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6349999999999997e-06 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084250+0000", 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096340109999999996 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.084263+0000", 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3292000000000001e-05 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087416+0000", 2024-09-13T11:32:55.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003152672 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:05.087447+0000", 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.1241999999999997e-05 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:32:55.288 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.910152504000003, 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:32:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:32:55.293 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:209 3.2 e38/30 3:6afe79ef:::smithi20438004-10 low_tier:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.171621496, 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016793596000000001, 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.092843+0000", 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4799999999999999e-07 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102771+0000", 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0099271120000000001 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102791+0000", 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0148000000000001e-05 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102801+0000", 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8560000000000002e-06 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102898+0000", 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7528000000000005e-05 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102915+0000", 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6915999999999999e-05 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109590+0000", 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066751579999999996 2024-09-13T11:32:55.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109637+0000", 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6529999999999997e-05 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.161621582999999, 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:32:55.298 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:32:55.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:32:55.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.149403989, 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:32:55.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:207 3.2 e38/30 3:59a396eb:::smithi20438004-8 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.142654284999999, 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025976583000000001, 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121810+0000", 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121811+0000", 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2799999999999996e-07 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128273+0000", 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064622129999999996 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128294+0000", 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1364000000000001e-05 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128312+0000", 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8002000000000001e-05 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128443+0000", 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013052699999999999 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128487+0000", 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4338999999999997e-05 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147644+0000", 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019156341 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147787+0000", 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014326900000000001 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:220 3.2 e38/30 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:55.311 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.128986654999999, 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013667876000000001, 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135478+0000", 2024-09-13T11:32:55.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135479+0000", 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.175e-06 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139134+0000", 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036550240000000002 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139152+0000", 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8136999999999999e-05 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139168+0000", 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5951e-05 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139254+0000", 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5691000000000003e-05 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139278+0000", 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4148e-05 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149117+0000", 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098384820000000008 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149146+0000", 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9268000000000002e-05 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.315 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:233 3.2 e38/30 3:53d0adee:::smithi20438004-34 low_tier:head v 38'7, mlcod=38'5)", 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.111506135999999, 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023237916000000001, 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152958+0000", 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.152960+0000", 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161956+0000", 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0089955509999999992 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161973+0000", 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7350000000000002e-05 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161984+0000", 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.11e-05 2024-09-13T11:32:55.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162047+0000", 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2677999999999999e-05 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162075+0000", 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8501e-05 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176139+0000", 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014063512 2024-09-13T11:32:55.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176196+0000", 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7661000000000002e-05 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:240 3.2 e38/30 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.098466411, 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028057044, 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.320 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.165998+0000", 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166000+0000", 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179000+0000", 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013000523999999999 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179014+0000", 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4143e-05 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179023+0000", 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5969999999999992e-06 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179083+0000", 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9911000000000003e-05 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179110+0000", 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7251999999999999e-05 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014884616999999999 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.194055+0000", 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:246 3.2 e38/30 3:68241bcb:::smithi20438004-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'6)", 2024-09-13T11:32:55.324 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.076954782000001, 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013298119000000001, 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187510+0000", 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.187511+0000", 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000003e-07 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197044+0000", 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095328879999999998 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197061+0000", 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7413000000000001e-05 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197072+0000", 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1078e-05 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197146+0000", 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4449999999999994e-05 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197176+0000", 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9584e-05 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.200753+0000", 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035767020000000002 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.200808+0000", 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5120000000000001e-05 2024-09-13T11:32:55.328 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:244 3.2 e38/30 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'6)", 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.067353871000002, 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023733269000000001, 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197111+0000", 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204547+0000", 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074358269999999999 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204567+0000", 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0196000000000001e-05 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204579+0000", 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2071e-05 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204679+0000", 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9702000000000002e-05 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204727+0000", 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7929000000000001e-05 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220809+0000", 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016082012999999999 2024-09-13T11:32:55.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220844+0000", 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4959999999999997e-05 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:248 3.2 e38/30 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'11, mlcod=38'8)", 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.059852663000001, 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016257252999999999, 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.333 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204612+0000", 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.204613+0000", 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5300000000000003e-07 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210186+0000", 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005573789 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210202+0000", 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5424999999999999e-05 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210215+0000", 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3393e-05 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210285+0000", 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9295000000000007e-05 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210305+0000", 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0557999999999999e-05 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220848+0000", 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010542463 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220869+0000", 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1577e-05 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.337 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:247 3.2 e38/30 3:4f19c0b8:::smithi20438004-48 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.054221517999999, 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016148066999999999, 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210243+0000", 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.210244+0000", 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4099999999999997e-07 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217679+0000", 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0074353470000000001 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217694+0000", 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.455e-05 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217702+0000", 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5580000000000001e-06 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217760+0000", 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7788000000000003e-05 2024-09-13T11:32:55.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.217786+0000", 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5803999999999999e-05 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226348+0000", 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085624180000000005 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226391+0000", 2024-09-13T11:32:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2660999999999997e-05 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1351 4.d 4:b7c9e12a:::smithi20438004-12:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:35.791077+0000", 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.473387816999999, 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022759601000000001, 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:55.342 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1351 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791077+0000", 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791077+0000", 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791079+0000", 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5280000000000002e-06 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791090+0000", 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0507e-05 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791112+0000", 2024-09-13T11:32:55.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2364000000000001e-05 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791125+0000", 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2434e-05 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791200+0000", 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5482999999999995e-05 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.791297+0000", 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7058999999999993e-05 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:35.813836+0000", 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022539225999999999 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.841766+0000", 2024-09-13T11:32:55.346 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.422698061999998, 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019390853999999999, 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.841766+0000", 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.841766+0000", 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.841767+0000", 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1400000000000001e-07 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855528+0000", 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013761379000000001 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855543+0000", 2024-09-13T11:32:55.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5065999999999999e-05 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855552+0000", 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8459999999999997e-06 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855603+0000", 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1294000000000001e-05 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855621+0000", 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7482999999999998e-05 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.861105+0000", 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054846870000000002 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.861157+0000", 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1885000000000002e-05 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.350 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:1651 3.2 e45/30 3:720dc439:::2c9d97cd8bc343ec27e96488c4591170a3e62f503f5d642c2598a4d2866780e1:head v 45'112, mlcod=45'110)", 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:41.057748+0000", 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.206716520000001, 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013229449000000001, 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.057748+0000", 2024-09-13T11:32:55.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.057748+0000", 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.057751+0000", 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8729999999999999e-06 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062779+0000", 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050277100000000003 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062814+0000", 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5336000000000002e-05 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062845+0000", 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0969000000000001e-05 2024-09-13T11:32:55.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062906+0000", 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1548000000000001e-05 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.062925+0000", 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8669e-05 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.070940+0000", 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0080148179999999999 2024-09-13T11:32:55.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:41.070977+0000", 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7526e-05 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1679 4.4 4:2b094039:::smithi20438004-40:head [tier-flush] snapc 0=[] RETRY=3 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.9952798820000002, 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0158098280000001, 2024-09-13T11:32:55.355 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1679 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269187+0000", 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9479999999999998e-06 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269193+0000", 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.173e-06 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269209+0000", 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6039e-05 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269224+0000", 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5494000000000001e-05 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270767+0000", 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015428869999999999 2024-09-13T11:32:55.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270782+0000", 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.519e-05 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284573+0000", 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013791077000000001 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284717+0000", 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014369599999999999 2024-09-13T11:32:55.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284994+0000", 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.000277324 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.360 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:55.361 INFO:tasks.rados.rados.0.smithi204.stdout:1113: writing smithi20438004-8 from 1586353 to 2098738 tid 2 2024-09-13T11:32:55.362 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:32:55.363 INFO:tasks.rados.rados.0.smithi204.stdout:1113: writing smithi20438004-8 from 2749730 to 3294172 tid 3 2024-09-13T11:32:55.363 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:55.363 INFO:tasks.rados.rados.0.smithi204.stdout:1099: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-13T11:32:55.363 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:55.360+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:55.425 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1198826 2024-09-13T11:32:55.463 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:55.462+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:55.474 INFO:tasks.rados.rados.0.smithi204.stdout:1100: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-13T11:32:55.501 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2867208 2024-09-13T11:32:55.507 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:55.507 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:55.507 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:55.508 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:55.509 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 82745 2024-09-13T11:32:55.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:32:55.559 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:55.564 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:55.563+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:55.608 INFO:tasks.rados.rados.0.smithi204.stdout:1102: finishing set_chunk to oid 30 2024-09-13T11:32:55.609 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 49 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-13T11:32:55.609 INFO:tasks.rados.rados.0.smithi204.stdout:1103: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-13T11:32:55.631 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:55.631 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:32:55.631 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:32:55.631 INFO:tasks.thrashosds.thrasher:Killing osd 12, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:32:55.632 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-13T11:32:55.632 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:32:55.662 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:55.662 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:55.662 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:55.662 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:55.662 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:55.663 INFO:tasks.ceph.osd.12:Stopped 2024-09-13T11:32:55.663 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.12 2024-09-13T11:32:55.663 INFO:tasks.thrashosds.thrasher:remote for osd 12 is ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:32:55.664 INFO:tasks.thrashosds.thrasher:remote for exp osd 2 is ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:32:55.664 DEBUG:teuthology.orchestra.run.smithi154:> type ceph-objectstore-tool 2024-09-13T11:32:55.664 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2533429 2024-09-13T11:32:55.665 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:55.664+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:55.671 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:32:55.680 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-13T11:32:55.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-09-13T11:32:55.728 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1813602 2024-09-13T11:32:55.766 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:55.764+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:55.816 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:55.816 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:55.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:55.816 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:55.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.816 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.716659169, 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:32:55.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:32:55.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:55.821 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.704225023999999, 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.825 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.694271430000001, 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:224 3.3 e38/30 3:e7e3acce:::smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=38'1)", 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.669632184000001, 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010654663, 2024-09-13T11:32:55.830 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144199+0000", 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.144201+0000", 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1969999999999999e-06 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150234+0000", 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060323859999999998 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150248+0000", 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4584e-05 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150256+0000", 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1270000000000003e-06 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150306+0000", 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9633999999999999e-05 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150327+0000", 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0579999999999999e-05 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154794+0000", 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004467108 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154854+0000", 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0047000000000003e-05 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.834 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.663553622999999, 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:55.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.644730303999999, 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.839 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:32:55.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:55.843 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.631210955, 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.621570622, 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.611265121999999, 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.852 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:32:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.60087918, 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:32:55.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:1480 3.3 e44/41 3:fc8d920c:::574e186947520a2389cbaeea8ba1176542b7ebf6bc2d1924ea8fea4e9a7674ef:head v 44'83, mlcod=44'81)", 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.100870+0000", 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.712961196999998, 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087291580000000007, 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.861 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.100870+0000", 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.100870+0000", 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.100872+0000", 2024-09-13T11:32:55.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5370000000000001e-06 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104077+0000", 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032057600000000002 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104088+0000", 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0220999999999999e-05 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104097+0000", 2024-09-13T11:32:55.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3000000000000007e-06 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104183+0000", 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5964999999999994e-05 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.104203+0000", 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0202000000000001e-05 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.109556+0000", 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0053534350000000001 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.109599+0000", 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:30 3.3 3:d93f0134:::e72e863a8da8d1de65cb61afe4c91921fe19461674a2d93807a62898d9604a30:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.269104+0000", 2024-09-13T11:32:55.865 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5447269410000004, 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010740975, 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269104+0000", 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269104+0000", 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269106+0000", 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.511e-06 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269527+0000", 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00042104799999999998 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269538+0000", 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1103e-05 2024-09-13T11:32:55.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269563+0000", 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4896999999999999e-05 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269705+0000", 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014246800000000001 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269715+0000", 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0077e-05 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.276019+0000", 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0063039480000000002 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.276224+0000", 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020479800000000001 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.276440+0000", 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021594600000000001 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278103+0000", 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001663168 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279530+0000", 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014263399999999999 2024-09-13T11:32:55.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279663+0000", 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00013359999999999999 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279728+0000", 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4580999999999997e-05 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279845+0000", 2024-09-13T11:32:55.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011749 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:33 3.3 3:eed165f9:::a6a31c61839f2616f379c5a86f65f5702175d12a1af122f7ffd409b19031e165:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.269582+0000", 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5442489960000003, 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010471223999999999, 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269582+0000", 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269582+0000", 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269583+0000", 2024-09-13T11:32:55.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4300000000000003e-07 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270289+0000", 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00070559599999999996 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270297+0000", 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270304+0000", 2024-09-13T11:32:55.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1720000000000001e-06 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270490+0000", 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018597500000000001 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270499+0000", 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4020000000000008e-06 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.276902+0000", 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064026350000000003 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277011+0000", 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010938900000000001 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277182+0000", 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000171316 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279538+0000", 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023558440000000002 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279600+0000", 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1959999999999996e-05 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279893+0000", 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029317600000000003 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279950+0000", 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6935000000000003e-05 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280053+0000", 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103151 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.878 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:34 3.3 3:ce7ec0f0:::302c6a3bdf6d249958d305a37c5742c0de5d7cad722f2dabd19f570d2f795527:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270390+0000", 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5434415819999998, 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010169920000000001, 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270390+0000", 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270390+0000", 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270391+0000", 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2900000000000002e-07 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270451+0000", 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0609999999999997e-05 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270460+0000", 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5900000000000008e-06 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270467+0000", 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2919999999999998e-06 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270524+0000", 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6900000000000001e-05 2024-09-13T11:32:55.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270531+0000", 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5359999999999998e-06 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277461+0000", 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069305629999999998 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277589+0000", 2024-09-13T11:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000128081 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277762+0000", 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017332 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280068+0000", 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023053209999999999 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280120+0000", 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2339000000000002e-05 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280422+0000", 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000301503 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280456+0000", 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3853999999999998e-05 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280560+0000", 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104082 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:36 3.3 3:e2816a99:::d4bb749789c90c884ffd20ac62f804c7a6b80502d14432fa627e1239249ee2da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5433517810000001, 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013305613000000001, 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.885 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270480+0000", 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2399999999999998e-07 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270655+0000", 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017481300000000001 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270683+0000", 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8340999999999999e-05 2024-09-13T11:32:55.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270693+0000", 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271029+0000", 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033640200000000002 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271038+0000", 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278355+0000", 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073166510000000004 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278481+0000", 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125955 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278657+0000", 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176816 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280615+0000", 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019576810000000002 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283538+0000", 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029229030000000001 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283592+0000", 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4366000000000001e-05 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283671+0000", 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8337000000000006e-05 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283785+0000", 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114412 2024-09-13T11:32:55.891 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:40 3.3 3:c7a12847:::768a08ae37e9bf19369a517906ebd209c1f2b8ad3684596d644da7d48b42114d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5431086749999992, 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013347114, 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270839+0000", 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115369 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270847+0000", 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270854+0000", 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:32:55.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271073+0000", 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021964400000000001 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271080+0000", 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7909999999999999e-06 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278928+0000", 2024-09-13T11:32:55.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078479819999999999 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279058+0000", 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130102 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279300+0000", 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024166300000000001 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280641+0000", 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013409229999999999 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283545+0000", 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002904401 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283927+0000", 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038193199999999997 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283964+0000", 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7117999999999999e-05 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284070+0000", 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010545899999999999 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.898 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:41 3.3 3:e9fa9a15:::3c65c6c9770b7a29cd5a9b6dc5b0b39ab23cb2a64bfb19442efcb911d255e067:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5429650620000004, 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013499547000000001, 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270867+0000", 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270893+0000", 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6057000000000001e-05 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270901+0000", 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270915+0000", 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4450000000000001e-05 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271104+0000", 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018904600000000001 2024-09-13T11:32:55.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271111+0000", 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6039999999999998e-06 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280674+0000", 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095629140000000005 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280811+0000", 2024-09-13T11:32:55.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136802 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281027+0000", 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000216617 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283802+0000", 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027746540000000001 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283884+0000", 2024-09-13T11:32:55.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2167000000000001e-05 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284178+0000", 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029347899999999997 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284246+0000", 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8609999999999995e-05 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284366+0000", 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011950799999999999 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:43 3.3 3:dc8816b1:::1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5429043670000002, 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013665096999999999, 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.905 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:32:55.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270933+0000", 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2410000000000003e-06 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270939+0000", 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5810000000000002e-06 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270945+0000", 2024-09-13T11:32:55.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271134+0000", 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000189161 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271140+0000", 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2999999999999998e-06 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281407+0000", 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010266226 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281549+0000", 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142583 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281737+0000", 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018791500000000001 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283810+0000", 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002072571 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284127+0000", 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000317516 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284417+0000", 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029000899999999998 2024-09-13T11:32:55.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284480+0000", 2024-09-13T11:32:55.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3136e-05 2024-09-13T11:32:55.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284592+0000", 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111712 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.10.22:39 3.3 3:c55f8587:::a2f2c330500f1b1d28c41e464817565848095218a89d32db0b731691bbe2b77b:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.5207085980000006, 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011011337, 2024-09-13T11:32:55.913 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293124+0000", 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5319999999999999e-06 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293132+0000", 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293143+0000", 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1477999999999999e-05 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293152+0000", 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8289999999999997e-06 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298197+0000", 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050456870000000001 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298273+0000", 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5799999999999999e-05 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298497+0000", 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000223618 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.299173+0000", 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00067642199999999998 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.299376+0000", 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020231700000000001 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304021+0000", 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004645003 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304068+0000", 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7139999999999999e-05 2024-09-13T11:32:55.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304134+0000", 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6203000000000003e-05 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2003 3.3 3:cd1157fc:::84107b35dec3bd49dc7100dbbfbae69c8e7aee4cb078460f18fcc3852b3346d1:head [write 0~216576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:50.856956+0000", 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.9568751799999999, 2024-09-13T11:32:55.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076454310000000003, 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2003 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.856956+0000", 2024-09-13T11:32:55.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.856956+0000", 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.856959+0000", 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5339999999999998e-06 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.857039+0000", 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0672000000000003e-05 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.857065+0000", 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5371000000000002e-05 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.857077+0000", 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2206e-05 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.857152+0000", 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5588999999999997e-05 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.857160+0000", 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7249999999999997e-06 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.860418+0000", 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.003258024 2024-09-13T11:32:55.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.860499+0000", 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1069999999999995e-05 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.860592+0000", 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2231000000000005e-05 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.863513+0000", 2024-09-13T11:32:55.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029211749999999998 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.863922+0000", 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040970299999999998 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.864511+0000", 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00058831400000000005 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.864545+0000", 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4538e-05 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.864602+0000", 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6279000000000001e-05 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:55.926 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:55.928 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1104: finishing set_chunk to oid 45 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 311808 length: 974336 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1099: done (15 left) 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1100: done (14 left) 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1102: done (13 left) 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1103: done (12 left) 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1104: done (11 left) 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1114: set_chunk oid 13 target oid 2024-09-13T11:32:55.930 INFO:tasks.rados.rados.0.smithi204.stdout:1114: set_chunk oid 13 offset: 724480 length: 233984 target oid 6c7aa39de4b80c368ded941b8d3c63445d6213686dc26f01ca415aa0f5f14490 offset: 0 snap -1 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1115: read oid 19 snap -1 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1115: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1116: read oid 18 snap -1 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1116: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1117: read oid 30 snap -1 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1117: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1118: tier_flush oid 22 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1118: tier_flush oid 22 snap -1 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1105: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:55.931 INFO:tasks.rados.rados.0.smithi204.stdout:1106: expect (ObjNum 176 snap 0 seq_num 176) 2024-09-13T11:32:55.932 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:55.926+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:55.932 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 781596 2024-09-13T11:32:56.028 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:56.027+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:56.036 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:56.036 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:32:56.037 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:56.043 INFO:teuthology.orchestra.run.smithi154.stdout:4.4 2024-09-13T11:32:56.043 INFO:teuthology.orchestra.run.smithi154.stdout:3.3 2024-09-13T11:32:56.043 INFO:teuthology.orchestra.run.smithi154.stdout:2.5 2024-09-13T11:32:56.045 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:32:56.062 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1981137 2024-09-13T11:32:56.094 INFO:tasks.rados.rados.0.smithi204.stdout:1107: finishing set_chunk to oid 34 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 212480 length: 2402304 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 0 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:1108: got (22) Invalid argument 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:1109: finishing write tid 1 to smithi20438004-12 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:1109: finishing write tid 2 to smithi20438004-12 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:1110: finishing set_chunk to oid 39 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:1110: got expected ENOENT (src dne) 2024-09-13T11:32:56.095 INFO:tasks.rados.rados.0.smithi204.stdout:1109: finishing write tid 3 to smithi20438004-12 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1109: finishing write tid 4 to smithi20438004-12 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1109: finishing write tid 5 to smithi20438004-12 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1109: oid 12 updating version 0 to 60 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1109: oid 12 version 60 is already newer than 59 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1109: oid 12 version 60 is already newer than 58 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1109: oid 12 version 60 is already newer than 57 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 60 (ObjNum 194 snap 0 seq_num 194) dirty exists 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1109: left oid 12 (ObjNum 194 snap 0 seq_num 194) 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1111: finishing write tid 1 to smithi20438004-46 2024-09-13T11:32:56.096 INFO:tasks.rados.rados.0.smithi204.stdout:1111: finishing write tid 2 to smithi20438004-46 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1111: finishing write tid 3 to smithi20438004-46 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1105: done (15 left) 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1106: done (14 left) 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1107: done (13 left) 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1108: done (12 left) 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1109: done (11 left) 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1110: done (10 left) 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1119: write oid 7 current snap is 0 2024-09-13T11:32:56.097 INFO:tasks.rados.rados.0.smithi204.stdout:1119: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-13T11:32:56.100 INFO:tasks.rados.rados.0.smithi204.stdout:1119: writing smithi20438004-7 from 510979 to 1043857 tid 1 2024-09-13T11:32:56.106 INFO:tasks.rados.rados.0.smithi204.stdout:1119: writing smithi20438004-7 from 1528482 to 2300878 tid 2 2024-09-13T11:32:56.112 INFO:tasks.rados.rados.0.smithi204.stdout:1119: writing smithi20438004-7 from 3060800 to 3310979 tid 3 2024-09-13T11:32:56.112 INFO:tasks.rados.rados.0.smithi204.stdout:1120: write oid 28 current snap is 0 2024-09-13T11:32:56.112 INFO:tasks.rados.rados.0.smithi204.stdout:1111: finishing write tid 4 to smithi20438004-46 2024-09-13T11:32:56.112 INFO:tasks.rados.rados.0.smithi204.stdout:1120: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-13T11:32:56.118 INFO:tasks.rados.rados.0.smithi204.stdout:1120: writing smithi20438004-28 from 527786 to 1252266 tid 1 2024-09-13T11:32:56.123 INFO:tasks.rados.rados.0.smithi204.stdout:1120: writing smithi20438004-28 from 1703317 to 2219371 tid 2 2024-09-13T11:32:56.129 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:56.127+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:56.130 INFO:tasks.rados.rados.0.smithi204.stdout:1120: writing smithi20438004-28 from 2804576 to 3239611 tid 3 2024-09-13T11:32:56.130 INFO:tasks.rados.rados.0.smithi204.stdout:1121: set_chunk oid 49 target oid 2024-09-13T11:32:56.130 INFO:tasks.rados.rados.0.smithi204.stdout:1121: set_chunk oid 49 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1111: finishing write tid 5 to smithi20438004-46 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1111: oid 46 updating version 0 to 56 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1111: oid 46 version 56 is already newer than 55 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1111: oid 46 version 56 is already newer than 54 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1111: oid 46 updating version 56 to 57 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 57 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1111: left oid 46 (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1112: finishing set_chunk to oid 2 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 65 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1113: finishing write tid 1 to smithi20438004-8 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1113: finishing write tid 2 to smithi20438004-8 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1113: finishing write tid 3 to smithi20438004-8 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1113: finishing write tid 4 to smithi20438004-8 2024-09-13T11:32:56.131 INFO:tasks.rados.rados.0.smithi204.stdout:1113: finishing write tid 5 to smithi20438004-8 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1113: oid 8 updating version 0 to 24 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1113: oid 8 version 24 is already newer than 22 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1113: oid 8 version 24 is already newer than 23 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1113: oid 8 version 24 is already newer than 21 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 24 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1113: left oid 8 (ObjNum 196 snap 0 seq_num 196) 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1122: read oid 10 snap -1 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1122: expect deleted 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1111: done (13 left) 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1112: done (12 left) 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1113: done (11 left) 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1123: write (excl) oid 23 current snap is 0 2024-09-13T11:32:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1123: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-13T11:32:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1123: writing smithi20438004-23 from 544593 to 1344322 tid 1 2024-09-13T11:32:56.144 INFO:tasks.rados.rados.0.smithi204.stdout:1123: writing smithi20438004-23 from 1761799 to 2537864 tid 2 2024-09-13T11:32:56.148 INFO:tasks.rados.rados.0.smithi204.stdout:1123: writing smithi20438004-23 from 2948352 to 3344593 tid 3 2024-09-13T11:32:56.148 INFO:tasks.rados.rados.0.smithi204.stdout:1124: read oid 39 snap -1 2024-09-13T11:32:56.148 INFO:tasks.rados.rados.0.smithi204.stdout:1124: expect deleted 2024-09-13T11:32:56.148 INFO:tasks.rados.rados.0.smithi204.stdout:1125: write oid 11 current snap is 0 2024-09-13T11:32:56.149 INFO:tasks.rados.rados.0.smithi204.stdout:1114: finishing set_chunk to oid 13 2024-09-13T11:32:56.149 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 47 (ObjNum 159 snap 0 seq_num 159) dirty exists 2024-09-13T11:32:56.149 INFO:tasks.rados.rados.0.smithi204.stdout:1115: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:32:56.152 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:32:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:32:56.152 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:32:56.152 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:56.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:32:56.245 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:56.244+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:56.273 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1715413 2024-09-13T11:32:56.310 INFO:tasks.rados.rados.0.smithi204.stdout:1120: finishing write tid 1 to smithi20438004-28 2024-09-13T11:32:56.310 INFO:tasks.rados.rados.0.smithi204.stdout:1120: finishing write tid 2 to smithi20438004-28 2024-09-13T11:32:56.310 INFO:tasks.rados.rados.0.smithi204.stdout:1125: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-13T11:32:56.316 INFO:tasks.rados.rados.0.smithi204.stdout:1125: writing smithi20438004-11 from 561400 to 1036378 tid 1 2024-09-13T11:32:56.321 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:32:56.321 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.227686984000002, 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:32:56.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:32:56.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:32:56.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.217412416999998, 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.330 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:32:56.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.204744516000002, 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:32:56.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:32:56.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:32:56.339 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.198134669000002, 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.340 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.189757639, 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:32:56.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.170495592000002, 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010735344000000001, 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.349 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148571+0000", 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154390+0000", 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058191809999999997 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154404+0000", 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4584e-05 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154415+0000", 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0661e-05 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154478+0000", 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2528999999999997e-05 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154502+0000", 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4460999999999998e-05 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159243+0000", 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047404259999999998 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159305+0000", 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2582999999999994e-05 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:32:56.353 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.164626460000001, 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:56.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:32:56.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.155725182000001, 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.358 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:32:56.362 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.148127459000001, 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.366 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.142723530000001, 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:32:56.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.13009984, 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:32:56.371 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:32:56.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.375 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.115920389999999, 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.107702222, 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:32:56.380 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:32:56.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:32:56.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:32:56.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:32:56.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:32:56.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.076878819000001, 2024-09-13T11:32:56.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:56.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.393 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:298 4.c 4:3a6b347b:::smithi20438004-49:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242839+0000", 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.076226237, 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010413649000000001, 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 298 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242839+0000", 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242839+0000", 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5799999999999998e-07 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242844+0000", 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3989999999999997e-06 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242862+0000", 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8201000000000001e-05 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242869+0000", 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7120000000000003e-06 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242917+0000", 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7948999999999998e-05 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242994+0000", 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6650999999999995e-05 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,12", 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249456+0000", 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0064623129999999999 2024-09-13T11:32:56.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250061+0000", 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00060531400000000003 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250256+0000", 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019423699999999999 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253177+0000", 2024-09-13T11:32:56.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029211020000000001 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253216+0000", 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9613000000000002e-05 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253253+0000", 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6399999999999997e-05 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.930164032, 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.400 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.403 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 20.224991979999999, 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.404 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:32:56.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:32:56.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.477039609999999, 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013151014000000001, 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842045+0000", 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865e-05 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848182+0000", 2024-09-13T11:32:56.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061372370000000002 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848205+0000", 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.298e-05 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848218+0000", 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.348e-05 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848300+0000", 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1766000000000003e-05 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848327+0000", 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6446999999999999e-05 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855119+0000", 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067926490000000004 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855177+0000", 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7590000000000003e-05 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1458 4.e 4:7984cf84:::smithi20438004-46:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 19.291427557999999, 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012793834, 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.413 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1458 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:32:56.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027640+0000", 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.046e-06 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027651+0000", 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.102e-05 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027669+0000", 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027681+0000", 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2177999999999999e-05 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027762+0000", 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0331e-05 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027868+0000", 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106443 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.040432+0000", 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012563632999999999 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.417 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1787 4.e 4:7022fc62:::smithi20438004-6:head [read 188928~1265152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.0453348669999993, 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013752110999999999, 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1787 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273731+0000", 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273734+0000", 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3510000000000002e-06 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273741+0000", 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273787+0000", 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6198000000000001e-05 2024-09-13T11:32:56.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274432+0000", 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064478300000000001 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274461+0000", 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9051999999999999e-05 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281751+0000", 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007289963 2024-09-13T11:32:56.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281984+0000", 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023257099999999999 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.287483+0000", 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054989829999999998 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:32:56.422 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:32:56.423 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:32:56.424 INFO:tasks.rados.rados.0.smithi204.stdout:1125: writing smithi20438004-11 from 1820281 to 2340004 tid 2 2024-09-13T11:32:56.424 INFO:tasks.rados.rados.0.smithi204.stdout:1125: writing smithi20438004-11 from 2859422 to 3317001 tid 3 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1114: done (13 left) 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1115: done (12 left) 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1126: read oid 20 snap -1 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1126: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1127: write oid 45 current snap is 0 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1121: finishing set_chunk to oid 49 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1121: got expected ENOENT (src dne) 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1120: finishing write tid 3 to smithi20438004-28 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1120: finishing write tid 4 to smithi20438004-28 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1120: finishing write tid 5 to smithi20438004-28 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1120: oid 28 updating version 0 to 53 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1120: oid 28 updating version 53 to 56 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1120: oid 28 version 56 is already newer than 54 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:1120: oid 28 version 56 is already newer than 55 2024-09-13T11:32:56.425 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 56 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-13T11:32:56.426 INFO:tasks.rados.rados.0.smithi204.stdout:1120: left oid 28 (ObjNum 198 snap 0 seq_num 198) 2024-09-13T11:32:56.426 INFO:tasks.rados.rados.0.smithi204.stdout:1127: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-13T11:32:56.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1127: writing smithi20438004-45 from 578207 to 1128434 tid 1 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1127: writing smithi20438004-45 from 1762410 to 2542144 tid 2 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1127: writing smithi20438004-45 from 3286845 to 3378207 tid 3 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1120: done (13 left) 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1121: done (12 left) 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1122: done (11 left) 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1128: write oid 32 current snap is 0 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1128: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1128: writing smithi20438004-32 from 595014 to 1220490 tid 1 2024-09-13T11:32:56.428 INFO:tasks.rados.rados.0.smithi204.stdout:1128: writing smithi20438004-32 from 1820892 to 2344284 tid 2 2024-09-13T11:32:56.429 INFO:tasks.rados.rados.0.smithi204.stdout:1128: writing smithi20438004-32 from 2797915 to 3394391 tid 3 2024-09-13T11:32:56.429 INFO:tasks.rados.rados.0.smithi204.stdout:1129: set_chunk oid 21 target oid 2024-09-13T11:32:56.429 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:56.423+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:56.524 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:56.523+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:56.546 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:56.546 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:56.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:32:56.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 3.3 --file /var/log/ceph/exp.3.3.2 2024-09-13T11:32:56.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:32:56.656 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:56.654+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:56.665 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:56.665 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:56.665 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:56.665 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:56.665 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:56.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:32:56.726 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:56.724+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:56.826 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:56.826 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'23)", 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.044245816, 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005604375, 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779709+0000", 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.63e-07 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779721+0000", 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.204e-05 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779728+0000", 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4150000000000002e-06 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779732+0000", 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6839999999999998e-06 2024-09-13T11:32:56.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779761+0000", 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9405999999999999e-05 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779773+0000", 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1398e-05 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785264+0000", 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054913089999999998 2024-09-13T11:32:56.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785313+0000", 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8659999999999998e-05 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:44 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'26, mlcod=24'23)", 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.043675729, 2024-09-13T11:32:56.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055622880000000003, 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780278+0000", 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780279+0000", 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780365+0000", 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6462e-05 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780372+0000", 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1980000000000004e-06 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.780376+0000", 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3280000000000002e-06 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784152+0000", 2024-09-13T11:32:56.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037767650000000001 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.784173+0000", 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0899000000000001e-05 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785814+0000", 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001640561 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785841+0000", 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6628e-05 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.086774318000003, 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.840 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'105, mlcod=24'100)", 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.086210715, 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061709499999999997, 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737744+0000", 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7700000000000004e-07 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737749+0000", 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737756+0000", 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7649999999999996e-06 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737760+0000", 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6650000000000001e-06 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742613+0000", 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048528909999999998 2024-09-13T11:32:56.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742629+0000", 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6481e-05 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743902+0000", 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012723630000000001 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743914+0000", 2024-09-13T11:32:56.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2663999999999999e-05 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:140 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'106, mlcod=25'101)", 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.085574332, 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005547064, 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738380+0000", 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738382+0000", 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5909999999999998e-06 2024-09-13T11:32:56.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738386+0000", 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5240000000000001e-06 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738392+0000", 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8660000000000002e-06 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.738396+0000", 2024-09-13T11:32:56.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6739999999999996e-06 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742884+0000", 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004487441 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742896+0000", 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1866e-05 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743915+0000", 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010193229999999999 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743927+0000", 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1779e-05 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.849 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.346848691000002, 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:32:56.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:32:56.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:56.858 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.341790133, 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:32:56.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:32:56.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:32:56.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.341366311000002, 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.866 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:32:56.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:32:56.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.873 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.341348267000001, 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.874 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:32:56.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:32:56.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.879 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.625107495000002, 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.880 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:32:56.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:32:56.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:32:56.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.611266705999999, 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.887 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:32:56.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:32:56.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:32:56.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:32:56.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.512298732000001, 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:56.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:32:56.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.902 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.b 51/50 [PushOp(4:da92e835:::smithi20438004-50:head, version: 38'19, data_included: [2616802~223548], data_size: 223548, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:da92e835:::smithi20438004-50:head@38'19, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2616802, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.511488784999999, 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055783109999999999, 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312466+0000", 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7499999999999999e-07 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312658+0000", 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019212100000000001 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312681+0000", 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3059999999999999e-05 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312694+0000", 2024-09-13T11:32:56.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2408999999999999e-05 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312765+0000", 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1755000000000007e-05 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312775+0000", 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2879999999999998e-06 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.318043+0000", 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052689030000000001 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1751 4.9 4:9441befd:::smithi20438004-8:head [write 420208~530210,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.438385952000001, 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060468199999999996, 2024-09-13T11:32:56.907 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1751 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385569+0000", 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.426e-06 2024-09-13T11:32:56.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386470+0000", 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090069800000000002 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386485+0000", 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5225000000000001e-05 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386495+0000", 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386574+0000", 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8911000000000004e-05 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386698+0000", 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012372800000000001 2024-09-13T11:32:56.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386913+0000", 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021553499999999999 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.388673+0000", 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017596210000000001 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.389582+0000", 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090943100000000004 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391260+0000", 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00167782 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391570+0000", 2024-09-13T11:32:56.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030968499999999997 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391615+0000", 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5151999999999997e-05 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1752 4.9 4:9441befd:::smithi20438004-8:head [write 1527054~766050] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:56.914 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.430589794999999, 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062506259999999996, 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1752 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393365+0000", 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0720000000000001e-06 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394108+0000", 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074250399999999995 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394120+0000", 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1990999999999999e-05 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394125+0000", 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7270000000000004e-06 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394182+0000", 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6181000000000003e-05 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394248+0000", 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6848999999999996e-05 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394333+0000", 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4970999999999997e-05 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396066+0000", 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001732546 2024-09-13T11:32:56.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397394+0000", 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013282680000000001 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399415+0000", 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002021063 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399585+0000", 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016999699999999999 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399615+0000", 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9456999999999999e-05 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1753 4.9 4:9441befd:::smithi20438004-8:head [setxattr _header (55) in=7b,truncate 2420208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.429812708, 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055775579999999998, 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1753 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394142+0000", 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7999999999999995e-07 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394147+0000", 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2120000000000002e-06 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394154+0000", 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6980000000000001e-06 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394160+0000", 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395493+0000", 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00133299 2024-09-13T11:32:56.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395574+0000", 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0944000000000003e-05 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395686+0000", 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112049 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396457+0000", 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077122900000000001 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397581+0000", 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011240989999999999 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399655+0000", 2024-09-13T11:32:56.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002074046 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399684+0000", 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8924999999999998e-05 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399719+0000", 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4459000000000003e-05 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.928 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1754 4.9 4:9441befd:::smithi20438004-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.429782232000001, 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057025110000000004, 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.929 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1754 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2300000000000001e-07 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394175+0000", 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5189999999999999e-06 2024-09-13T11:32:56.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394181+0000", 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6089999999999997e-06 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394187+0000", 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7400000000000001e-06 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395928+0000", 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017413470000000001 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395983+0000", 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399771+0000", 2024-09-13T11:32:56.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003787786 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399814+0000", 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2755e-05 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399874+0000", 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0350999999999999e-05 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.934 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1959 4.e e53/51 4:7984cf84:::smithi20438004-46:head v 53'41, mlcod=52'38)", 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.6355374630000004, 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057232109999999997, 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:32:56.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0599999999999999e-07 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188422+0000", 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9409999999999998e-06 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188434+0000", 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2002e-05 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188441+0000", 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0090000000000001e-06 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188712+0000", 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027109600000000002 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188724+0000", 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1790999999999999e-05 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194092+0000", 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053680450000000001 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194140+0000", 2024-09-13T11:32:56.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7521000000000001e-05 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2036 4.e e54/51 4:76a15ccf:::smithi20438004-1:head v 54'46, mlcod=54'44)", 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2459889579999999, 2024-09-13T11:32:56.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056625620000000003, 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577966+0000", 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4200000000000005e-07 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578870+0000", 2024-09-13T11:32:56.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090414100000000002 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578880+0000", 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0501e-05 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578886+0000", 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.356e-06 2024-09-13T11:32:56.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578950+0000", 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578968+0000", 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7935999999999999e-05 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583569+0000", 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046012680000000004 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583628+0000", 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8473000000000001e-05 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2293 4.9 4:9441befd:::smithi20438004-8:head [write 1586353~512385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.552081018, 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066015969999999998, 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2293 2024-09-13T11:32:56.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271874+0000", 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.497e-06 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272359+0000", 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048419700000000001 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272399+0000", 2024-09-13T11:32:56.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0754000000000003e-05 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272410+0000", 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272470+0000", 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0603999999999998e-05 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272575+0000", 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104901 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272731+0000", 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015523300000000001 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.274392+0000", 2024-09-13T11:32:56.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016616070000000001 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.276887+0000", 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024952360000000001 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278386+0000", 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014986540000000001 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278439+0000", 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2624999999999998e-05 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278475+0000", 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5812999999999997e-05 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:56.953 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:56.953 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:56.954 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:56.825+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:56.955 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:56.953+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:56.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:32:56.957 INFO:teuthology.orchestra.run.smithi154.stderr:Exporting 3.3 info 3.3( v 39'53 (39'50,39'53] local-lis/les=30/31 n=11 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) 2024-09-13T11:32:56.957 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:c8f80c84:::bc036ea868a6b2c601148e4e87101fdc0cdd45a73131aa4e7533f4932f45b24d:head# 2024-09-13T11:32:56.957 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:32:56.957 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:dd3ad6bf:::4ec2dd001b7153a0bb00c76a8320dec150c3eb87238ccdd9135be5e948aee2d9:head# 2024-09-13T11:32:56.957 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:de2fec76:::smithi20438004-12 low_tier:head# 2024-09-13T11:32:56.958 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:e7e3acce:::smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:32:56.958 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:f1414861:::194f870459df717b3a43948c3b56bc35cac5308afcac3cd76a1a7e1a04723236:head# 2024-09-13T11:32:56.958 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:32:56.958 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:f5bbb099:::9b0f46e51af81c905599e26964e04a497e64c0e02a1d69859f9c292a656388d5:head# 2024-09-13T11:32:56.958 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:32:56.958 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:32:56.958 INFO:teuthology.orchestra.run.smithi154.stderr:Read #3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:32:56.960 INFO:teuthology.orchestra.run.smithi154.stderr:Export successful 2024-09-13T11:32:57.054 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:57.053+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:57.065 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:57.065 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:32:57.065 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:57.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:32:57.155 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:57.154+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:57.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:32:57.181 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:32:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:32:57.181 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:32:57.181 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:57.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:32:57.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 3.3 2024-09-13T11:32:57.271 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:32:57.272 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:57.270+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:57.295 INFO:tasks.rados.rados.0.smithi204.stdout:1129: set_chunk oid 21 offset: 2283008 length: 595968 target oid 01485695156c783754228fffee3e12487d04219290d0491fdefc817f45e18ad6 offset: 0 snap -1 2024-09-13T11:32:57.295 INFO:tasks.rados.rados.0.smithi204.stdout:1130: write oid 36 current snap is 0 2024-09-13T11:32:57.295 INFO:tasks.rados.rados.0.smithi204.stdout:1130: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-13T11:32:57.303 INFO:tasks.rados.rados.0.smithi204.stdout:1130: writing smithi20438004-36 from 611821 to 1312546 tid 1 2024-09-13T11:32:57.311 INFO:tasks.rados.rados.0.smithi204.stdout:1130: writing smithi20438004-36 from 1879374 to 2662777 tid 2 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout:1130: writing smithi20438004-36 from 3341691 to 3411821 tid 3 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout:1131: read oid 35 snap -1 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout:1131: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout:1132: read oid 8 snap -1 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout:1132: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout:1123: finishing write tid 1 to smithi20438004-23 2024-09-13T11:32:57.317 INFO:tasks.rados.rados.0.smithi204.stdout:1123: finishing write tid 2 to smithi20438004-23 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: finishing write tid 3 to smithi20438004-23 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1124: done (15 left) 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1133: write oid 44 current snap is 0 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: finishing write tid 4 to smithi20438004-23 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: finishing write tid 5 to smithi20438004-23 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: oid 23 updating version 0 to 72 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: oid 23 updating version 72 to 73 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: oid 23 updating version 73 to 74 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: oid 23 updating version 74 to 75 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 75 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1123: left oid 23 (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:32:57.318 INFO:tasks.rados.rados.0.smithi204.stdout:1133: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-13T11:32:57.325 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:57.326 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:57.326 INFO:tasks.rados.rados.0.smithi204.stdout:1133: writing smithi20438004-44 from 628628 to 1404602 tid 1 2024-09-13T11:32:57.332 INFO:tasks.rados.rados.0.smithi204.stdout:1133: writing smithi20438004-44 from 1937856 to 2464917 tid 2 2024-09-13T11:32:57.339 INFO:tasks.rados.rados.0.smithi204.stdout:1133: writing smithi20438004-44 from 3252761 to 3428628 tid 3 2024-09-13T11:32:57.339 INFO:tasks.rados.rados.0.smithi204.stdout:1123: done (15 left) 2024-09-13T11:32:57.339 INFO:tasks.rados.rados.0.smithi204.stdout:1134: read oid 48 snap -1 2024-09-13T11:32:57.339 INFO:tasks.rados.rados.0.smithi204.stdout:1134: expect (ObjNum 188 snap 0 seq_num 188) 2024-09-13T11:32:57.339 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:57.339 INFO:tasks.rados.rados.0.smithi204.stdout:1126: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:32:57.339 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:57.348 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:57.349 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:57.349 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:57.357 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:57.355+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:57.358 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:32:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:32:57.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.833240289999999, 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.359 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:32:57.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:32:57.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:32:57.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:26 4.8 4:1cf41451:::smithi20438004-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.827708877999999, 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083581760000000001, 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.368 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 26 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528196+0000", 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.048e-06 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528601+0000", 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040467299999999998 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528612+0000", 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.171e-05 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528620+0000", 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1140000000000006e-06 2024-09-13T11:32:57.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532030+0000", 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034098779999999999 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532128+0000", 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7910000000000003e-05 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532280+0000", 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015168700000000001 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533907+0000", 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016271 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533914+0000", 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2590000000000003e-06 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536475+0000", 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025609259999999998 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536517+0000", 2024-09-13T11:32:57.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.163e-05 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536553+0000", 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6241000000000001e-05 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.375 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:27 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.827271746000001, 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082102310000000001, 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 27 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.51e-07 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528635+0000", 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8399999999999999e-06 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528639+0000", 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528642+0000", 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.19e-06 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533286+0000", 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046437859999999996 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533371+0000", 2024-09-13T11:32:57.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4900000000000004e-05 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533475+0000", 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104285 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534168+0000", 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069284800000000003 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534400+0000", 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023182699999999999 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536780+0000", 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023795880000000002 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536809+0000", 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9156e-05 2024-09-13T11:32:57.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536842+0000", 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3546e-05 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:28 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.827246776999999, 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083368629999999999, 2024-09-13T11:32:57.382 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 28 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528659+0000", 2024-09-13T11:32:57.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528664+0000", 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.42e-06 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528675+0000", 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1447999999999999e-05 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533704+0000", 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050284769999999999 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533777+0000", 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3675000000000002e-05 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536883+0000", 2024-09-13T11:32:57.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031062820000000001 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536929+0000", 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5237000000000001e-05 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536994+0000", 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5152000000000002e-05 2024-09-13T11:32:57.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.144302149000001, 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.388 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:57.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:32:57.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:32:57.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.135959951, 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.396 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.403 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.115601947999998, 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.404 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:32:57.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.107801189, 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:32:57.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:32:57.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:32:57.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.558294381, 2024-09-13T11:32:57.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:797 4.8 4:1cf41451:::smithi20438004-7:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.544931817999998, 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086937119999999993, 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.425 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 797 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810974+0000", 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7010000000000001e-06 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811111+0000", 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013712699999999999 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811123+0000", 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1854e-05 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811131+0000", 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3979999999999999e-06 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813131+0000", 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019997740000000002 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813239+0000", 2024-09-13T11:32:57.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108378 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813421+0000", 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018227099999999999 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814991+0000", 2024-09-13T11:32:57.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015690599999999999 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816465+0000", 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001474784 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819575+0000", 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031099769999999999 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819626+0000", 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0609999999999998e-05 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819666+0000", 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9777999999999998e-05 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:798 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.544758192, 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087603679999999993, 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.433 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 798 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811150+0000", 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811157+0000", 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811163+0000", 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813998+0000", 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002835382 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814069+0000", 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1099000000000005e-05 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814212+0000", 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142647 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816475+0000", 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022629799999999999 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816520+0000", 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5099999999999998e-05 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819739+0000", 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032186570000000002 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819820+0000", 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1527999999999996e-05 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819906+0000", 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5845000000000002e-05 2024-09-13T11:32:57.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:799 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.544716852000001, 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089993910000000007, 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.440 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 799 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811188+0000", 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811192+0000", 2024-09-13T11:32:57.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.566e-06 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811202+0000", 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4790000000000002e-06 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811209+0000", 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814715+0000", 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035061150000000002 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814798+0000", 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2854000000000003e-05 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819962+0000", 2024-09-13T11:32:57.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051643419999999997 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820067+0000", 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010525 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820186+0000", 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118886 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:805 4.8 4:1d367c64:::smithi20438004-45:head [read 0~2739508,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.544639512, 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082479100000000007, 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.446 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 805 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811265+0000", 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5400000000000002e-07 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811266+0000", 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.939e-06 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811270+0000", 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4319999999999999e-06 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811274+0000", 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4150000000000001e-06 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818160+0000", 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068854700000000003 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818409+0000", 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024941299999999998 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819512+0000", 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011029869999999999 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.596105846, 2024-09-13T11:32:57.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:32:57.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:32:57.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.460 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1484 4.8 4:1251e6cc:::smithi20438004-13:head [sparse-read 0~2885135] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.245697188000001, 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074627620000000004, 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1484 2024-09-13T11:32:57.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110207+0000", 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2800000000000002e-07 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110209+0000", 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.855e-06 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110213+0000", 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9299999999999996e-06 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110216+0000", 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2389999999999998e-06 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.116449+0000", 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062328000000000001 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.116522+0000", 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3187000000000002e-05 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117669+0000", 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001147323 2024-09-13T11:32:57.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1485 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~3966419,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.245638750000001, 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01527358, 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1485 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2300000000000002e-07 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110269+0000", 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110274+0000", 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2349999999999999e-06 2024-09-13T11:32:57.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110278+0000", 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6100000000000002e-06 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117718+0000", 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074403120000000001 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117821+0000", 2024-09-13T11:32:57.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103501 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.125539+0000", 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077171590000000003 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1636 4.8 4:1d367c64:::smithi20438004-45:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.355471583, 2024-09-13T11:32:57.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091050230000000003, 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1636 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000433+0000", 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1200000000000003e-07 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000436+0000", 2024-09-13T11:32:57.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000443+0000", 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8619999999999997e-06 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000447+0000", 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8800000000000001e-06 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.001976+0000", 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001529774 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.002101+0000", 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124824 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009537+0000", 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074360850000000003 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.474 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1637 4.8 4:1d367c64:::smithi20438004-45:head [sparse-read 0~2368645] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.355449054000001, 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010163363, 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.475 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1637 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000457+0000", 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000460+0000", 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3720000000000001e-06 2024-09-13T11:32:57.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000463+0000", 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.225e-06 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009568+0000", 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091047630000000001 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009626+0000", 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8142e-05 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.010618+0000", 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099181000000000009 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.479 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1749 4.8 4:1d367c64:::smithi20438004-45:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.568202888, 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49694346099999998, 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1749 2024-09-13T11:32:57.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787703+0000", 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6840000000000001e-06 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787708+0000", 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787721+0000", 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2405000000000001e-05 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787728+0000", 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.391e-06 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787780+0000", 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1897000000000001e-05 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787845+0000", 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5184000000000005e-05 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284644+0000", 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49679929099999998 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2310 4.18 e60/59 4:1cf41451:::smithi20438004-7:head v 60'49, mlcod=0'0)", 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.071207216000000004, 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075680649999999997, 2024-09-13T11:32:57.485 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284698+0000", 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.043e-06 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285649+0000", 2024-09-13T11:32:57.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095142200000000005 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285665+0000", 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6317999999999999e-05 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285674+0000", 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1969999999999996e-06 2024-09-13T11:32:57.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285731+0000", 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7611000000000001e-05 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285747+0000", 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6211000000000001e-05 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.292214+0000", 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064668670000000003 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.292265+0000", 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0396000000000003e-05 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:32:57.491 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:32:57.491 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:32:57.492 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:57.492 INFO:tasks.rados.rados.0.smithi204.stdout:1128: finishing write tid 1 to smithi20438004-32 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: finishing write tid 2 to smithi20438004-32 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: finishing write tid 3 to smithi20438004-32 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: finishing write tid 4 to smithi20438004-32 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: finishing write tid 5 to smithi20438004-32 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: oid 32 updating version 0 to 42 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: oid 32 version 42 is already newer than 39 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: oid 32 version 42 is already newer than 41 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: oid 32 version 42 is already newer than 40 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 42 (ObjNum 202 snap 0 seq_num 202) dirty exists 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1128: left oid 32 (ObjNum 202 snap 0 seq_num 202) 2024-09-13T11:32:57.493 INFO:tasks.rados.rados.0.smithi204.stdout:1117: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:32:57.494 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:57.494 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:57.494 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:57.491+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:57.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:32:57.495 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:57.496 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:57.496 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:57.496 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:57.497 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:57.497 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:57.497 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:57.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:57.498 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:57.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:57.498 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:57.499 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:57.499 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:57.499 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:57.499 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:57.500 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:57.500 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:32:57.517 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:57.517 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:57.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:57.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:57.552 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:57.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:57.555 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:57.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:57.572 INFO:tasks.rados.rados.0.smithi204.stdout:1119: finishing write tid 1 to smithi20438004-7 2024-09-13T11:32:57.573 INFO:tasks.rados.rados.0.smithi204.stdout:1116: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:32:57.573 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:57.596 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:57.594+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:57.608 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:57.608 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:57.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:32:57.694 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:57.692+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:57.708 INFO:tasks.rados.rados.0.smithi204.stdout:1125: finishing write tid 1 to smithi20438004-11 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1129: finishing set_chunk to oid 21 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 22 (ObjNum 176 snap 0 seq_num 176) dirty exists 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1125: finishing write tid 2 to smithi20438004-11 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1125: finishing write tid 3 to smithi20438004-11 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1071: finishing write tid 1 to smithi20438004-24 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1116: done (15 left) 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1117: done (14 left) 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1126: done (13 left) 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1128: done (12 left) 2024-09-13T11:32:57.709 INFO:tasks.rados.rados.0.smithi204.stdout:1129: done (11 left) 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1135: tier_flush oid 33 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1135: tier_flush oid 33 snap -1 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1071: finishing write tid 2 to smithi20438004-24 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1071: finishing write tid 3 to smithi20438004-24 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1119: finishing write tid 2 to smithi20438004-7 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1071: finishing write tid 4 to smithi20438004-24 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1071: oid 24 updating version 0 to 84 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1071: oid 24 version 84 is already newer than 83 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1071: oid 24 updating version 84 to 85 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 85 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-09-13T11:32:57.710 INFO:tasks.rados.rados.0.smithi204.stdout:1071: left oid 24 (ObjNum 182 snap 0 seq_num 182) 2024-09-13T11:32:57.711 INFO:tasks.rados.rados.0.smithi204.stdout:1088: expect (ObjNum 167 snap 0 seq_num 167) 2024-09-13T11:32:57.711 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:57.765 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:57.766 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:57.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:32:57.795 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:32:57.793+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:32:57.827 INFO:tasks.rados.rados.0.smithi204.stdout:1119: finishing write tid 3 to smithi20438004-7 2024-09-13T11:32:57.827 INFO:tasks.rados.rados.0.smithi204.stdout:1119: finishing write tid 4 to smithi20438004-7 2024-09-13T11:32:57.827 INFO:tasks.rados.rados.0.smithi204.stdout:1119: finishing write tid 5 to smithi20438004-7 2024-09-13T11:32:57.827 INFO:tasks.rados.rados.0.smithi204.stdout:1119: oid 7 updating version 0 to 51 2024-09-13T11:32:57.827 INFO:tasks.rados.rados.0.smithi204.stdout:1119: oid 7 version 51 is already newer than 49 2024-09-13T11:32:57.827 INFO:tasks.rados.rados.0.smithi204.stdout:1119: oid 7 updating version 51 to 52 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:1119: oid 7 version 52 is already newer than 50 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 52 (ObjNum 197 snap 0 seq_num 197) dirty exists 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:1119: left oid 7 (ObjNum 197 snap 0 seq_num 197) 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:1125: finishing write tid 4 to smithi20438004-11 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:1127: finishing write tid 1 to smithi20438004-45 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:1125: finishing write tid 5 to smithi20438004-11 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:1125: oid 11 updating version 0 to 83 2024-09-13T11:32:57.828 INFO:tasks.rados.rados.0.smithi204.stdout:1125: oid 11 updating version 83 to 85 2024-09-13T11:32:57.829 INFO:tasks.rados.rados.0.smithi204.stdout:1125: oid 11 updating version 85 to 86 2024-09-13T11:32:57.830 INFO:tasks.rados.rados.0.smithi204.stdout:1125: oid 11 version 86 is already newer than 84 2024-09-13T11:32:57.831 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 86 (ObjNum 200 snap 0 seq_num 200) dirty exists 2024-09-13T11:32:57.831 INFO:tasks.rados.rados.0.smithi204.stdout:1125: left oid 11 (ObjNum 200 snap 0 seq_num 200) 2024-09-13T11:32:57.831 INFO:tasks.rados.rados.0.smithi204.stdout:1127: finishing write tid 2 to smithi20438004-45 2024-09-13T11:32:57.831 INFO:tasks.rados.rados.0.smithi204.stdout:1127: finishing write tid 3 to smithi20438004-45 2024-09-13T11:32:57.831 INFO:tasks.rados.rados.0.smithi204.stdout:1071: done (11 left) 2024-09-13T11:32:57.831 INFO:tasks.rados.rados.0.smithi204.stdout:1088: done (10 left) 2024-09-13T11:32:57.831 INFO:tasks.rados.rados.0.smithi204.stdout:1119: done (9 left) 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1125: done (8 left) 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1136: read oid 23 snap -1 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1136: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1137: write (excl) oid 26 current snap is 0 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1127: finishing write tid 4 to smithi20438004-45 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1130: finishing write tid 1 to smithi20438004-36 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1137: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-13T11:32:57.832 INFO:tasks.rados.rados.0.smithi204.stdout:1137: writing smithi20438004-26 from 645435 to 1096658 tid 1 2024-09-13T11:32:57.840 INFO:tasks.rados.rados.0.smithi204.stdout:1137: writing smithi20438004-26 from 1879985 to 2667057 tid 2 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1137: writing smithi20438004-26 from 3280184 to 3445435 tid 3 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1138: tier_flush oid 16 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1138: tier_flush oid 16 snap -1 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1127: finishing write tid 5 to smithi20438004-45 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1127: oid 45 updating version 0 to 55 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1127: oid 45 version 55 is already newer than 53 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1127: oid 45 version 55 is already newer than 54 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1127: oid 45 updating version 55 to 56 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 56 (ObjNum 201 snap 0 seq_num 201) dirty exists 2024-09-13T11:32:57.846 INFO:tasks.rados.rados.0.smithi204.stdout:1127: left oid 45 (ObjNum 201 snap 0 seq_num 201) 2024-09-13T11:32:57.847 INFO:tasks.rados.rados.0.smithi204.stdout:1127: done (10 left) 2024-09-13T11:32:57.847 INFO:tasks.rados.rados.0.smithi204.stdout:1139: tier_evict oid 40 2024-09-13T11:32:57.847 INFO:tasks.rados.rados.0.smithi204.stdout:1139: tier_evict oid 40 snap -1 2024-09-13T11:32:57.847 INFO:tasks.rados.rados.0.smithi204.stdout:1140: delete oid 37 current snap is 0 2024-09-13T11:32:57.847 INFO:tasks.rados.rados.0.smithi204.stdout:1131: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:32:57.883 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:57.883 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:57.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:32:57.895 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:32:57.893+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:32:57.996 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:32:57.995+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:32:58.009 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.009 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:32:58.031 INFO:tasks.rados.rados.0.smithi204.stdout:1130: finishing write tid 2 to smithi20438004-36 2024-09-13T11:32:58.031 INFO:tasks.rados.rados.0.smithi204.stdout:1130: finishing write tid 3 to smithi20438004-36 2024-09-13T11:32:58.031 INFO:tasks.rados.rados.0.smithi204.stdout:1130: finishing write tid 4 to smithi20438004-36 2024-09-13T11:32:58.031 INFO:tasks.rados.rados.0.smithi204.stdout:1131: done (11 left) 2024-09-13T11:32:58.031 INFO:tasks.rados.rados.0.smithi204.stdout:1140: done (10 left) 2024-09-13T11:32:58.031 INFO:tasks.rados.rados.0.smithi204.stdout:1141: write oid 10 current snap is 0 2024-09-13T11:32:58.031 INFO:tasks.rados.rados.0.smithi204.stdout:1130: finishing write tid 5 to smithi20438004-36 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:1130: oid 36 updating version 0 to 54 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:1130: oid 36 version 54 is already newer than 51 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:1130: oid 36 version 54 is already newer than 52 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:1130: oid 36 version 54 is already newer than 53 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 54 (ObjNum 203 snap 0 seq_num 203) dirty exists 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:1130: left oid 36 (ObjNum 203 snap 0 seq_num 203) 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:1133: finishing write tid 1 to smithi20438004-44 2024-09-13T11:32:58.032 INFO:tasks.rados.rados.0.smithi204.stdout:1132: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-13T11:32:58.096 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:58.095+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:58.097 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:58.121 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.122 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.130 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:32:58.197 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:32:58.195+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:32:58.240 INFO:tasks.rados.rados.0.smithi204.stdout:1133: finishing write tid 2 to smithi20438004-44 2024-09-13T11:32:58.240 INFO:tasks.rados.rados.0.smithi204.stdout:1133: finishing write tid 3 to smithi20438004-44 2024-09-13T11:32:58.241 INFO:tasks.rados.rados.0.smithi204.stdout:1133: finishing write tid 4 to smithi20438004-44 2024-09-13T11:32:58.241 INFO:tasks.rados.rados.0.smithi204.stdout:1133: finishing write tid 5 to smithi20438004-44 2024-09-13T11:32:58.241 INFO:tasks.rados.rados.0.smithi204.stdout:1133: oid 44 updating version 0 to 36 2024-09-13T11:32:58.241 INFO:tasks.rados.rados.0.smithi204.stdout:1133: oid 44 updating version 36 to 37 2024-09-13T11:32:58.241 INFO:tasks.rados.rados.0.smithi204.stdout:1133: oid 44 version 37 is already newer than 35 2024-09-13T11:32:58.241 INFO:tasks.rados.rados.0.smithi204.stdout:1133: oid 44 version 37 is already newer than 34 2024-09-13T11:32:58.242 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 37 (ObjNum 204 snap 0 seq_num 204) dirty exists 2024-09-13T11:32:58.242 INFO:tasks.rados.rados.0.smithi204.stdout:1133: left oid 44 (ObjNum 204 snap 0 seq_num 204) 2024-09-13T11:32:58.242 INFO:tasks.rados.rados.0.smithi204.stdout:1141: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-13T11:32:58.247 INFO:tasks.rados.rados.0.smithi204.stdout:1141: writing smithi20438004-10 from 679049 to 1397123 tid 1 2024-09-13T11:32:58.255 INFO:tasks.rados.rados.0.smithi204.stdout:1141: writing smithi20438004-10 from 2113302 to 2787690 tid 2 2024-09-13T11:32:58.259 INFO:tasks.rados.rados.0.smithi204.stdout:1141: writing smithi20438004-10 from 3335030 to 3479049 tid 3 2024-09-13T11:32:58.260 INFO:tasks.rados.rados.0.smithi204.stdout:1130: done (10 left) 2024-09-13T11:32:58.260 INFO:tasks.rados.rados.0.smithi204.stdout:1132: done (9 left) 2024-09-13T11:32:58.260 INFO:tasks.rados.rados.0.smithi204.stdout:1133: done (8 left) 2024-09-13T11:32:58.260 INFO:tasks.rados.rados.0.smithi204.stdout:1142: set_chunk oid 28 target oid 2024-09-13T11:32:58.273 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.273 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.275 INFO:tasks.rados.rados.0.smithi204.stdout:1142: set_chunk oid 28 offset: 1016832 length: 2281472 target oid 79d32138cfe2fb2d6dec43a692a4fe25fa388f405df069ef6c6843185a7a5dd5 offset: 0 snap -1 2024-09-13T11:32:58.275 INFO:tasks.rados.rados.0.smithi204.stdout:1143: tier_evict oid 24 2024-09-13T11:32:58.275 INFO:tasks.rados.rados.0.smithi204.stdout:1143: tier_evict oid 24 snap -1 2024-09-13T11:32:58.276 INFO:tasks.rados.rados.0.smithi204.stdout:1134: expect (ObjNum 188 snap 0 seq_num 188) 2024-09-13T11:32:58.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:32:58.309 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:58.325 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:58.324+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:58.375 INFO:tasks.rados.rados.0.smithi204.stdout:1135: got (0) Success 2024-09-13T11:32:58.375 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 184 snap 0 seq_num 184) dirty exists 2024-09-13T11:32:58.375 INFO:tasks.rados.rados.0.smithi204.stdout:1134: done (9 left) 2024-09-13T11:32:58.375 INFO:tasks.rados.rados.0.smithi204.stdout:1135: done (8 left) 2024-09-13T11:32:58.375 INFO:tasks.rados.rados.0.smithi204.stdout:1144: read oid 30 snap -1 2024-09-13T11:32:58.375 INFO:tasks.rados.rados.0.smithi204.stdout:1144: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:32:58.376 INFO:tasks.rados.rados.0.smithi204.stdout:1145: read oid 14 snap -1 2024-09-13T11:32:58.376 INFO:tasks.rados.rados.0.smithi204.stdout:1145: expect deleted 2024-09-13T11:32:58.376 INFO:tasks.rados.rados.0.smithi204.stdout:1146: write (excl) oid 11 current snap is 0 2024-09-13T11:32:58.376 INFO:tasks.rados.rados.0.smithi204.stdout:1137: finishing write tid 1 to smithi20438004-26 2024-09-13T11:32:58.376 INFO:tasks.rados.rados.0.smithi204.stdout:1136: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:32:58.385 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.385 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:32:58.434 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:32:58.433+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:32:58.485 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:58.500 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:32:58.498+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:32:58.536 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.536 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.545 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:32:58.594 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:58.600 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:32:58.599+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:32:58.648 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.648 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:32:58.667 INFO:teuthology.orchestra.run.smithi154.stdout: marking collection for removal 2024-09-13T11:32:58.667 INFO:teuthology.orchestra.run.smithi154.stdout:setting '_remove' omap key 2024-09-13T11:32:58.667 INFO:teuthology.orchestra.run.smithi154.stdout:finish_remove_pgs 3.3_head removing 3.3 2024-09-13T11:32:58.673 INFO:teuthology.orchestra.run.smithi154.stdout:Remove successful 2024-09-13T11:32:58.701 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:58.700+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: finishing write tid 2 to smithi20438004-26 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: finishing write tid 3 to smithi20438004-26 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: finishing write tid 4 to smithi20438004-26 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: finishing write tid 5 to smithi20438004-26 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: oid 26 updating version 0 to 45 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: oid 26 updating version 45 to 46 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: oid 26 version 46 is already newer than 44 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: oid 26 version 46 is already newer than 43 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 46 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-13T11:32:58.704 INFO:tasks.rados.rados.0.smithi204.stdout:1137: left oid 26 (ObjNum 205 snap 0 seq_num 205) 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1138: got (0) Success 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: finishing write tid 1 to smithi20438004-10 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: finishing write tid 2 to smithi20438004-10 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: finishing write tid 3 to smithi20438004-10 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: finishing write tid 4 to smithi20438004-10 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: finishing write tid 5 to smithi20438004-10 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: oid 10 updating version 0 to 102 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: oid 10 updating version 102 to 104 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: oid 10 version 104 is already newer than 103 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: oid 10 version 104 is already newer than 101 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 104 (ObjNum 207 snap 0 seq_num 207) dirty exists 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1141: left oid 10 (ObjNum 207 snap 0 seq_num 207) 2024-09-13T11:32:58.705 INFO:tasks.rados.rados.0.smithi204.stdout:1143: got (22) Invalid argument 2024-09-13T11:32:58.706 INFO:tasks.rados.rados.0.smithi204.stdout:1142: finishing set_chunk to oid 28 2024-09-13T11:32:58.706 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 56 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-13T11:32:58.706 INFO:tasks.rados.rados.0.smithi204.stdout:1139: got (22) Invalid argument 2024-09-13T11:32:58.706 INFO:tasks.rados.rados.0.smithi204.stdout:1118: got (0) Success 2024-09-13T11:32:58.706 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-09-13T11:32:58.706 INFO:tasks.rados.rados.0.smithi204.stdout:1146: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-13T11:32:58.711 INFO:tasks.rados.rados.0.smithi204.stdout:1146: writing smithi20438004-11 from 695856 to 1489179 tid 1 2024-09-13T11:32:58.718 INFO:tasks.rados.rados.0.smithi204.stdout:1146: writing smithi20438004-11 from 2171784 to 2706183 tid 2 2024-09-13T11:32:58.722 INFO:tasks.rados.rados.0.smithi204.stdout:1146: writing smithi20438004-11 from 3362453 to 3495856 tid 3 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1118: done (10 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1136: done (9 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1137: done (8 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1138: done (7 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1139: done (6 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1141: done (5 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1142: done (4 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1143: done (3 left) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1147: read oid 21 snap -1 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1147: expect (ObjNum 176 snap 0 seq_num 176) 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1148: write oid 39 current snap is 0 2024-09-13T11:32:58.723 INFO:tasks.rados.rados.0.smithi204.stdout:1144: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:32:58.746 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:58.763 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.763 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:32:58.777 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: finishing write tid 1 to smithi20438004-11 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: finishing write tid 2 to smithi20438004-11 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: finishing write tid 3 to smithi20438004-11 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: finishing write tid 4 to smithi20438004-11 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: finishing write tid 5 to smithi20438004-11 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: oid 11 updating version 0 to 88 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: oid 11 updating version 88 to 89 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: oid 11 version 89 is already newer than 87 2024-09-13T11:32:58.808 INFO:tasks.rados.rados.0.smithi204.stdout:1146: oid 11 updating version 89 to 90 2024-09-13T11:32:58.809 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 90 (ObjNum 208 snap 0 seq_num 208) dirty exists 2024-09-13T11:32:58.809 INFO:tasks.rados.rados.0.smithi204.stdout:1146: left oid 11 (ObjNum 208 snap 0 seq_num 208) 2024-09-13T11:32:58.809 INFO:tasks.rados.rados.0.smithi204.stdout:1147: expect (ObjNum 176 snap 0 seq_num 176) 2024-09-13T11:32:58.809 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:58.818 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:58.817+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:58.873 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:58.873 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:58.882 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:32:58.923 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:58.922+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:58.927 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1957538 2024-09-13T11:32:58.959 INFO:tasks.rados.rados.0.smithi204.stdout:1148: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-13T11:32:58.965 INFO:tasks.rados.rados.0.smithi204.stdout:1148: writing smithi20438004-39 from 712663 to 1181235 tid 1 2024-09-13T11:32:58.971 INFO:tasks.rados.rados.0.smithi204.stdout:1148: writing smithi20438004-39 from 1713913 to 2391970 tid 2 2024-09-13T11:32:58.977 INFO:tasks.rados.rados.0.smithi204.stdout:1148: writing smithi20438004-39 from 2873523 to 3465256 tid 3 2024-09-13T11:32:58.977 INFO:tasks.rados.rados.0.smithi204.stdout:1144: done (4 left) 2024-09-13T11:32:58.978 INFO:tasks.rados.rados.0.smithi204.stdout:1146: done (3 left) 2024-09-13T11:32:58.978 INFO:tasks.rados.rados.0.smithi204.stdout:1147: done (2 left) 2024-09-13T11:32:58.978 INFO:tasks.rados.rados.0.smithi204.stdout:1149: write oid 11 current snap is 0 2024-09-13T11:32:58.978 INFO:tasks.rados.rados.0.smithi204.stdout:1148: finishing write tid 1 to smithi20438004-39 2024-09-13T11:32:58.978 INFO:tasks.rados.rados.0.smithi204.stdout:1148: finishing write tid 2 to smithi20438004-39 2024-09-13T11:32:58.978 INFO:tasks.rados.rados.0.smithi204.stdout:1149: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-13T11:32:58.983 INFO:tasks.rados.rados.0.smithi204.stdout:1149: writing smithi20438004-11 from 729470 to 1273291 tid 1 2024-09-13T11:32:58.989 INFO:tasks.rados.rados.0.smithi204.stdout:1149: writing smithi20438004-11 from 1772395 to 2310463 tid 2 2024-09-13T11:32:58.996 INFO:tasks.rados.rados.0.smithi204.stdout:1149: writing smithi20438004-11 from 2900946 to 3303951 tid 3 2024-09-13T11:32:58.996 INFO:tasks.rados.rados.0.smithi204.stdout:1150: read oid 23 snap -1 2024-09-13T11:32:58.996 INFO:tasks.rados.rados.0.smithi204.stdout:1150: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: finishing write tid 3 to smithi20438004-39 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: finishing write tid 4 to smithi20438004-39 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: finishing write tid 5 to smithi20438004-39 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: oid 39 updating version 0 to 80 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: oid 39 version 80 is already newer than 77 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: oid 39 version 80 is already newer than 78 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: oid 39 version 80 is already newer than 79 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 80 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: left oid 39 (ObjNum 209 snap 0 seq_num 209) 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1149: finishing write tid 1 to smithi20438004-11 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1151: read oid 37 snap -1 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1151: expect deleted 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1148: done (4 left) 2024-09-13T11:32:58.997 INFO:tasks.rados.rados.0.smithi204.stdout:1152: tier_evict oid 35 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1152: tier_evict oid 35 snap -1 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1149: finishing write tid 2 to smithi20438004-11 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1153: read oid 47 snap -1 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1153: expect (ObjNum 158 snap 0 seq_num 158) 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1154: read oid 43 snap -1 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1154: expect (ObjNum 170 snap 0 seq_num 170) 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1155: tier_flush oid 10 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1155: tier_flush oid 10 snap -1 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1156: tier_flush oid 4 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1156: tier_flush oid 4 snap -1 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1157: write oid 13 current snap is 0 2024-09-13T11:32:58.998 INFO:tasks.rados.rados.0.smithi204.stdout:1157: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-13T11:32:59.003 INFO:tasks.rados.rados.0.smithi204.stdout:1157: writing smithi20438004-13 from 746277 to 1365347 tid 1 2024-09-13T11:32:59.004 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:59.002+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:59.011 INFO:tasks.rados.rados.0.smithi204.stdout:1157: writing smithi20438004-13 from 1830877 to 2512603 tid 2 2024-09-13T11:32:59.017 INFO:tasks.rados.rados.0.smithi204.stdout:1157: writing smithi20438004-13 from 2928369 to 3546277 tid 3 2024-09-13T11:32:59.017 INFO:tasks.rados.rados.0.smithi204.stdout:1158: tier_flush oid 2 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1158: tier_flush oid 2 snap -1 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1152: got (22) Invalid argument 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1152: done (10 left) 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1159: read oid 45 snap -1 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1159: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1160: read oid 29 snap -1 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1160: expect (ObjNum 167 snap 0 seq_num 167) 2024-09-13T11:32:59.018 INFO:tasks.rados.rados.0.smithi204.stdout:1161: set_chunk oid 6 target oid 2024-09-13T11:32:59.028 INFO:tasks.rados.rados.0.smithi204.stdout:1161: set_chunk oid 6 offset: 458752 length: 713728 target oid 55ef7dc67155414771152cd340b10738d08f3bfae29a7ca55de1d56e5a6e080a offset: 0 snap -1 2024-09-13T11:32:59.028 INFO:tasks.rados.rados.0.smithi204.stdout:1162: write (excl) oid 44 current snap is 0 2024-09-13T11:32:59.029 INFO:tasks.rados.rados.0.smithi204.stdout:1162: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-13T11:32:59.029 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.030 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.037 INFO:tasks.rados.rados.0.smithi204.stdout:1162: writing smithi20438004-44 from 763084 to 1457403 tid 1 2024-09-13T11:32:59.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:32:59.043 INFO:tasks.rados.rados.0.smithi204.stdout:1162: writing smithi20438004-44 from 1889359 to 2431096 tid 2 2024-09-13T11:32:59.050 INFO:tasks.rados.rados.0.smithi204.stdout:1162: writing smithi20438004-44 from 2955792 to 3497694 tid 3 2024-09-13T11:32:59.050 INFO:tasks.rados.rados.0.smithi204.stdout:1163: read oid 25 snap -1 2024-09-13T11:32:59.050 INFO:tasks.rados.rados.0.smithi204.stdout:1163: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:32:59.050 INFO:tasks.rados.rados.0.smithi204.stdout:1164: write (excl) oid 34 current snap is 0 2024-09-13T11:32:59.050 INFO:tasks.rados.rados.0.smithi204.stdout:1164: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-13T11:32:59.056 INFO:tasks.rados.rados.0.smithi204.stdout:1164: writing smithi20438004-34 from 779891 to 1265812 tid 1 2024-09-13T11:32:59.064 INFO:tasks.rados.rados.0.smithi204.stdout:1164: writing smithi20438004-34 from 1947841 to 2633236 tid 2 2024-09-13T11:32:59.069 INFO:tasks.rados.rados.0.smithi204.stdout:1164: writing smithi20438004-34 from 3383215 to 3579891 tid 3 2024-09-13T11:32:59.069 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:59.069 INFO:tasks.rados.rados.0.smithi204.stdout:1149: finishing write tid 3 to smithi20438004-11 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1149: finishing write tid 4 to smithi20438004-11 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1149: finishing write tid 5 to smithi20438004-11 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1149: oid 11 updating version 0 to 92 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1149: oid 11 updating version 92 to 93 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1149: oid 11 updating version 93 to 94 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1149: oid 11 version 94 is already newer than 91 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 94 (ObjNum 210 snap 0 seq_num 210) dirty exists 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1149: left oid 11 (ObjNum 210 snap 0 seq_num 210) 2024-09-13T11:32:59.070 INFO:tasks.rados.rados.0.smithi204.stdout:1153: expect (ObjNum 158 snap 0 seq_num 158) 2024-09-13T11:32:59.104 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:32:59.103+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:32:59.141 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.142 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:32:59.164 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:59.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-12 --journal-path /var/lib/ceph/osd/ceph-12/journal --op list-pgs 2024-09-13T11:32:59.210 INFO:tasks.rados.rados.0.smithi204.stdout:1150: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:32:59.227 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:32:59.226+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:32:59.253 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.253 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.261 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:32:59.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:32:59.306 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:32:59.305+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:32:59.362 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1925836 2024-09-13T11:32:59.364 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.365 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:32:59.407 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:32:59.405+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:32:59.477 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.478 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:32:59.508 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:59.507+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:59.533 INFO:tasks.rados.rados.0.smithi204.stdout:1154: expect (ObjNum 170 snap 0 seq_num 170) 2024-09-13T11:32:59.533 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:4.f 2024-09-13T11:32:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:4.c 2024-09-13T11:32:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:4.6 2024-09-13T11:32:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:4.3 2024-09-13T11:32:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:4.16 2024-09-13T11:32:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:4.19 2024-09-13T11:32:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:4.12 2024-09-13T11:32:59.538 INFO:teuthology.orchestra.run.smithi038.stdout:4.13 2024-09-13T11:32:59.538 INFO:teuthology.orchestra.run.smithi038.stdout:3.2 2024-09-13T11:32:59.538 INFO:teuthology.orchestra.run.smithi038.stdout:3.0 2024-09-13T11:32:59.538 INFO:teuthology.orchestra.run.smithi038.stdout:4.2 2024-09-13T11:32:59.538 INFO:teuthology.orchestra.run.smithi038.stdout:2.4 2024-09-13T11:32:59.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:32:59.590 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.590 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.592 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 558395 2024-09-13T11:32:59.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:32:59.608 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:59.607+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:59.641 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:32:59.684 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:32:59.685 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1157: finishing write tid 1 to smithi20438004-13 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1156: got (0) Success 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1149: done (15 left) 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1150: done (14 left) 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1153: done (13 left) 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1154: done (12 left) 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1156: done (11 left) 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1165: set_chunk oid 49 target oid 2024-09-13T11:32:59.698 INFO:tasks.rados.rados.0.smithi204.stdout:1165: set_chunk oid 49 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:59.699 INFO:tasks.rados.rados.0.smithi204.stdout:1166: read oid 8 snap -1 2024-09-13T11:32:59.699 INFO:tasks.rados.rados.0.smithi204.stdout:1166: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-13T11:32:59.699 INFO:tasks.rados.rados.0.smithi204.stdout:1167: write (excl) oid 38 current snap is 0 2024-09-13T11:32:59.699 INFO:tasks.rados.rados.0.smithi204.stdout:1157: finishing write tid 2 to smithi20438004-13 2024-09-13T11:32:59.699 INFO:tasks.rados.rados.0.smithi204.stdout:1157: finishing write tid 3 to smithi20438004-13 2024-09-13T11:32:59.699 INFO:tasks.rados.rados.0.smithi204.stdout:1167: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-13T11:32:59.703 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.704 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.704 INFO:tasks.rados.rados.0.smithi204.stdout:1167: writing smithi20438004-38 from 796698 to 1357868 tid 1 2024-09-13T11:32:59.705 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:32:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:32:59.705 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:32:59.709 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:32:59.708+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:32:59.711 INFO:tasks.rados.rados.0.smithi204.stdout:1167: writing smithi20438004-38 from 2006323 to 2551729 tid 2 2024-09-13T11:32:59.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:32:59.718 INFO:tasks.rados.rados.0.smithi204.stdout:1167: writing smithi20438004-38 from 3126991 to 3596698 tid 3 2024-09-13T11:32:59.718 INFO:tasks.rados.rados.0.smithi204.stdout:1168: set_chunk oid 27 target oid 2024-09-13T11:32:59.718 INFO:tasks.rados.rados.0.smithi204.stdout:1157: finishing write tid 4 to smithi20438004-13 2024-09-13T11:32:59.718 INFO:tasks.rados.rados.0.smithi204.stdout:1157: finishing write tid 5 to smithi20438004-13 2024-09-13T11:32:59.718 INFO:tasks.rados.rados.0.smithi204.stdout:1157: oid 13 updating version 0 to 52 2024-09-13T11:32:59.719 INFO:tasks.rados.rados.0.smithi204.stdout:1157: oid 13 version 52 is already newer than 50 2024-09-13T11:32:59.719 INFO:tasks.rados.rados.0.smithi204.stdout:1157: oid 13 version 52 is already newer than 51 2024-09-13T11:32:59.719 INFO:tasks.rados.rados.0.smithi204.stdout:1157: oid 13 updating version 52 to 53 2024-09-13T11:32:59.719 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 53 (ObjNum 211 snap 0 seq_num 211) dirty exists 2024-09-13T11:32:59.719 INFO:tasks.rados.rados.0.smithi204.stdout:1157: left oid 13 (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:1161: finishing set_chunk to oid 6 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 458752 length: 713728 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:1155: got (0) Success 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 207 snap 0 seq_num 207) dirty exists 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:1162: finishing write tid 1 to smithi20438004-44 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:1158: got (0) Success 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:1162: finishing write tid 2 to smithi20438004-44 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:1168: set_chunk oid 27 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:32:59.720 INFO:tasks.rados.rados.0.smithi204.stdout:1155: done (14 left) 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1157: done (13 left) 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1158: done (12 left) 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1161: done (11 left) 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1169: read oid 16 snap -1 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1169: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1170: read oid 22 snap -1 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1170: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-13T11:32:59.721 INFO:tasks.rados.rados.0.smithi204.stdout:1171: set_chunk oid 4 target oid 2024-09-13T11:32:59.725 INFO:tasks.rados.rados.0.smithi204.stdout:1171: set_chunk oid 4 offset: 307712 length: 335360 target oid 1885e65fb6f62bdb1cf898dd111e82fc24564bb53b9423b2df031f09a49a863b offset: 0 snap -1 2024-09-13T11:32:59.725 INFO:tasks.rados.rados.0.smithi204.stdout:1172: set_chunk oid 46 target oid 2024-09-13T11:32:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:32:59.726 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:32:59.735 INFO:tasks.rados.rados.0.smithi204.stdout:1172: set_chunk oid 46 offset: 280576 length: 1132032 target oid 1bf4fc59d961a272e9aac6574b96bb162beaa3c217c1fec1bc4fa408684103ad offset: 0 snap -1 2024-09-13T11:32:59.735 INFO:tasks.rados.rados.0.smithi204.stdout:1173: read oid 6 snap -1 2024-09-13T11:32:59.735 INFO:tasks.rados.rados.0.smithi204.stdout:1173: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:32:59.735 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:32:59.735 INFO:tasks.rados.rados.0.smithi204.stdout:1162: finishing write tid 3 to smithi20438004-44 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1162: finishing write tid 4 to smithi20438004-44 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1162: finishing write tid 5 to smithi20438004-44 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1162: oid 44 updating version 0 to 39 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1162: oid 44 updating version 39 to 42 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1162: oid 44 version 42 is already newer than 40 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1162: oid 44 version 42 is already newer than 41 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 42 (ObjNum 212 snap 0 seq_num 212) dirty exists 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1162: left oid 44 (ObjNum 212 snap 0 seq_num 212) 2024-09-13T11:32:59.736 INFO:tasks.rados.rados.0.smithi204.stdout:1163: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:32:59.743 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:32:59.743 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:32:59.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:32:59.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:32:59.778 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:32:59.778 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:32:59.794 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:32:59.795 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:32:59.810 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:32:59.809+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:32:59.811 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:32:59.811 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:32:59.818 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:32:59.818 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:32:59.825 INFO:tasks.thrashosds.thrasher:Moving pg 3.3 from osd.2 to osd.12 2024-09-13T11:32:59.825 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi154.front.sepia.ceph.com to ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:32:59.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2024-09-13T11:32:59.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:32:59.828 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:32:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:32:59.846 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:32:59.846 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:32:59.862 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:32:59.863 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:32:59.863 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:32:59.880 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:32:59.880 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:32:59.891 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2024-09-13T11:32:59.897 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:32:59.898 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:32:59.911 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:32:59.910+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:32:59.915 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:32:59.916 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:32:59.917 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-13T11:32:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:32:59.919 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:32:59.920 INFO:tasks.rados.rados.0.smithi204.stdout:1164: finishing write tid 1 to smithi20438004-34 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: finishing write tid 2 to smithi20438004-34 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: finishing write tid 3 to smithi20438004-34 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: finishing write tid 4 to smithi20438004-34 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: finishing write tid 5 to smithi20438004-34 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: oid 34 updating version 0 to 101 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: oid 34 updating version 101 to 103 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: oid 34 version 103 is already newer than 102 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: oid 34 version 103 is already newer than 100 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 103 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: left oid 34 (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1162: done (15 left) 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1163: done (14 left) 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1164: done (13 left) 2024-09-13T11:32:59.921 INFO:tasks.rados.rados.0.smithi204.stdout:1174: read oid 25 snap -1 2024-09-13T11:32:59.922 INFO:tasks.rados.rados.0.smithi204.stdout:1174: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:32:59.922 INFO:tasks.rados.rados.0.smithi204.stdout:1175: read oid 18 snap -1 2024-09-13T11:32:59.922 INFO:tasks.rados.rados.0.smithi204.stdout:1175: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:32:59.922 INFO:tasks.rados.rados.0.smithi204.stdout:1176: write (excl) oid 33 current snap is 0 2024-09-13T11:32:59.922 INFO:tasks.rados.rados.0.smithi204.stdout:1159: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-13T11:32:59.946 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.R7NQA6cXzl 2024-09-13T11:32:59.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /var/log/ceph/exp.3.3.2 /tmp/tmp.R7NQA6cXzl 2024-09-13T11:32:59.947 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1960991 2024-09-13T11:32:59.974 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:32:59.974 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:32:59.974 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:32:59.974 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:32:59.974 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:32:59.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:00.013 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:00.011+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:00.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.R7NQA6cXzl 2024-09-13T11:33:00.113 INFO:tasks.rados.rados.0.smithi204.stdout:1160: expect (ObjNum 167 snap 0 seq_num 167) 2024-09-13T11:33:00.114 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:00.112+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:00.130 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:254 4.0 4:04aeeab9:::smithi20438004-5:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 32.92657509, 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022108323999999999, 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.131 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 254 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:00.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201594+0000", 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201603+0000", 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.568e-06 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201619+0000", 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6540000000000001e-05 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201629+0000", 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3200000000000006e-06 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201685+0000", 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6078000000000001e-05 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201747+0000", 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2242999999999993e-05 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220103+0000", 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01835548 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220937+0000", 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083429299999999995 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221938+0000", 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010010710000000001 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223567+0000", 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001628846 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223638+0000", 2024-09-13T11:33:00.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1165000000000001e-05 2024-09-13T11:33:00.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223701+0000", 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3120999999999994e-05 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:268 4.f 4:f653a832:::smithi20438004-19:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 32.904209682999998, 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01043647, 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.138 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 268 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:33:00.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223960+0000", 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.641e-06 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223968+0000", 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1820000000000006e-06 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223982+0000", 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3671e-05 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223989+0000", 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224062+0000", 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2315000000000001e-05 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224166+0000", 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010401299999999999 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232681+0000", 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085151910000000001 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233615+0000", 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00093373600000000001 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234090+0000", 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000475699 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234332+0000", 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241895 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234362+0000", 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9889000000000001e-05 2024-09-13T11:33:00.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234395+0000", 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2471000000000001e-05 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:274 4.f 4:f3c80275:::smithi20438004-25:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.228974+0000", 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 32.899193359999998, 2024-09-13T11:33:00.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010225435999999999, 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 274 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228974+0000", 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228974+0000", 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228976+0000", 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4729999999999999e-06 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.228983+0000", 2024-09-13T11:33:00.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5229999999999998e-06 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229000+0000", 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6748e-05 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229009+0000", 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8480000000000007e-06 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229077+0000", 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7637000000000003e-05 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.229172+0000", 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5486e-05 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233860+0000", 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046882069999999998 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234569+0000", 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00070882900000000001 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.235208+0000", 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00063829299999999996 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239093+0000", 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038850719999999998 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239148+0000", 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5562000000000001e-05 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239200+0000", 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1758000000000001e-05 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.150 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:285 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 32.893590631000002, 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012191677, 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 285 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234578+0000", 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234583+0000", 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9110000000000001e-06 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234593+0000", 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234598+0000", 2024-09-13T11:33:00.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9509999999999997e-06 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234639+0000", 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1826999999999998e-05 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234688+0000", 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9039000000000003e-05 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:00.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239203+0000", 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045143739999999998 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239979+0000", 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077664100000000001 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.242634+0000", 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026544899999999998 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246683+0000", 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040491499999999996 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246723+0000", 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0429000000000002e-05 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246769+0000", 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5430999999999999e-05 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.156 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1715 4.f 4:fea8c5b2:::smithi20438004-39:head [write 769787~704463,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.859781691, 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020210576000000001, 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.157 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1715 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268388+0000", 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269172+0000", 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078478 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269184+0000", 2024-09-13T11:33:00.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.182e-05 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269194+0000", 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8570000000000007e-06 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269269+0000", 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4795000000000005e-05 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269283+0000", 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275426+0000", 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061428050000000003 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275540+0000", 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011432199999999999 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275658+0000", 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118071 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278252+0000", 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594224 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279142+0000", 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088964600000000001 2024-09-13T11:33:00.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288321+0000", 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091785600000000005 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288363+0000", 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2957000000000002e-05 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288492+0000", 2024-09-13T11:33:00.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012896899999999999 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288597+0000", 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010429000000000001 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1716 4.f 4:fea8c5b2:::smithi20438004-39:head [write 2267961~101826] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:00.164 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.858954834, 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019610079999999998, 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1716 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269214+0000", 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9699999999999995e-07 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269305+0000", 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1330999999999997e-05 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269314+0000", 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9209999999999998e-06 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269321+0000", 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:33:00.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269363+0000", 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2388999999999997e-05 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269372+0000", 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277175+0000", 2024-09-13T11:33:00.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078032580000000004 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277247+0000", 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1773999999999994e-05 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277354+0000", 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010704000000000001 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288308+0000", 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010953353000000001 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288328+0000", 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0353e-05 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288640+0000", 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031178999999999999 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288706+0000", 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6749999999999996e-05 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288765+0000", 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8876999999999999e-05 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288823+0000", 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7806999999999997e-05 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.171 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1717 4.f 4:fea8c5b2:::smithi20438004-39:head [setxattr _header (56) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.858830572, 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019744788999999999, 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.172 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1717 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269338+0000", 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269342+0000", 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.924e-06 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269349+0000", 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9619999999999996e-06 2024-09-13T11:33:00.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269355+0000", 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1210000000000003e-06 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269399+0000", 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4360000000000002e-05 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:00.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5570000000000001e-06 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278293+0000", 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088867549999999997 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278372+0000", 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9561999999999995e-05 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278492+0000", 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119654 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288335+0000", 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098426840000000008 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288672+0000", 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033785899999999998 2024-09-13T11:33:00.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288863+0000", 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019105599999999999 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288904+0000", 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0738000000000002e-05 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288949+0000", 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4443000000000001e-05 2024-09-13T11:33:00.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289082+0000", 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013346199999999999 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1718 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.858801098000001, 2024-09-13T11:33:00.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019946973, 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1718 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8999999999999996e-07 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269373+0000", 2024-09-13T11:33:00.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8629999999999996e-06 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269379+0000", 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4979999999999997e-06 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269383+0000", 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7260000000000002e-06 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269443+0000", 2024-09-13T11:33:00.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9616e-05 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269451+0000", 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279055+0000", 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009604586 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:00.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279104+0000", 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8131999999999999e-05 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289119+0000", 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010015616 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289177+0000", 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8060999999999999e-05 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289314+0000", 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000136525 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1732 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.858776603000001, 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017069699000000001, 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.185 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1732 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269392+0000", 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269395+0000", 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-06 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269400+0000", 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.198e-06 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9410000000000004e-06 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269514+0000", 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108124 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269524+0000", 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279170+0000", 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096457209999999995 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279248+0000", 2024-09-13T11:33:00.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8147999999999998e-05 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286461+0000", 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072129789999999996 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1733 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.858733626999999, 2024-09-13T11:33:00.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0187283, 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1733 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269435+0000", 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7800000000000001e-07 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269438+0000", 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.535e-06 2024-09-13T11:33:00.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269447+0000", 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6559999999999998e-06 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269452+0000", 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.972e-06 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269552+0000", 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9820000000000003e-05 2024-09-13T11:33:00.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269556+0000", 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286496+0000", 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016940182000000002 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286550+0000", 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3640999999999997e-05 2024-09-13T11:33:00.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288163+0000", 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016126720000000001 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1734 4.f 4:f653a832:::smithi20438004-19:head [read 1042153~2571319,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.858667726, 2024-09-13T11:33:00.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021098286000000001, 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1734 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269501+0000", 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269504+0000", 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.58e-06 2024-09-13T11:33:00.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269511+0000", 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6200000000000001e-06 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269517+0000", 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7030000000000003e-06 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269580+0000", 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:33:00.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269583+0000", 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5020000000000001e-06 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289341+0000", 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019757429 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289385+0000", 2024-09-13T11:33:00.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3939000000000001e-05 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.290598+0000", 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012136130000000001 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.12.0:167 4.6 e51/31 4:6978b14d:::smithi20438004-23:head v 51'52, mlcod=45'50)", 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 12.936553383, 2024-09-13T11:33:00.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013003298, 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:00.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191616+0000", 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0270000000000001e-06 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196708+0000", 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050925720000000001 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196728+0000", 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9616999999999999e-05 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196738+0000", 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8330000000000006e-06 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196817+0000", 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9225000000000001e-05 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196841+0000", 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3972e-05 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204573+0000", 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077323629999999999 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204618+0000", 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4688999999999999e-05 2024-09-13T11:33:00.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1770 4.0 4:001d90c2:::smithi20438004-28:head [read 0~2218524,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 12.858378764999999, 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011933050000000001, 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1770 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269791+0000", 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269795+0000", 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269803+0000", 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269810+0000", 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.246e-06 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273037+0000", 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032276700000000002 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273048+0000", 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0511e-05 2024-09-13T11:33:00.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.277997+0000", 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049497539999999998 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278150+0000", 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152406 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281722+0000", 2024-09-13T11:33:00.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00357224 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1771 4.0 4:001d90c2:::smithi20438004-28:head [read 633380~1585144,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 12.858348761, 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013022562, 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.211 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.211 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1771 2024-09-13T11:33:00.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.211 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:00.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269820+0000", 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4000000000000002e-07 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269821+0000", 2024-09-13T11:33:00.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9269999999999999e-06 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269825+0000", 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8789999999999996e-06 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269829+0000", 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4740000000000001e-06 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273144+0000", 2024-09-13T11:33:00.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033155459999999999 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273155+0000", 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.062e-05 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282280+0000", 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091252620000000003 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282403+0000", 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012240499999999999 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282842+0000", 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043900900000000002 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1772 4.0 4:001d90c2:::smithi20438004-28:head [sparse-read 2038646~179878] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:00.216 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 12.858335316, 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013177762000000001, 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1772 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9499999999999998e-07 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269834+0000", 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269837+0000", 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.005e-06 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269843+0000", 2024-09-13T11:33:00.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6389999999999997e-06 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273188+0000", 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033454840000000001 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273196+0000", 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0830000000000004e-06 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282874+0000", 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096776810000000005 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282924+0000", 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0095000000000001e-05 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.283010+0000", 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6216000000000003e-05 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.221 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2324 4.16 4:6978b14d:::smithi20438004-23:head [write 544593~799729,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.991087335, 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14427186, 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.222 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2324 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7100000000000001e-07 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137660+0000", 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057879000000000003 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137671+0000", 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0518e-05 2024-09-13T11:33:00.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137675+0000", 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137754+0000", 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8492000000000007e-05 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137765+0000", 2024-09-13T11:33:00.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1027e-05 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273433+0000", 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13566771999999999 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273569+0000", 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013616999999999999 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:00.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273699+0000", 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130142 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281227+0000", 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075284699999999998 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281259+0000", 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1610999999999999e-05 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281298+0000", 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9138999999999999e-05 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281352+0000", 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2325 4.16 4:6978b14d:::smithi20438004-23:head [write 1761799~776065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:00.228 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.9844220190000001, 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13997515799999999, 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2325 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143748+0000", 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000001e-06 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144385+0000", 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000637125 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144396+0000", 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0633e-05 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144401+0000", 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2469999999999999e-06 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144457+0000", 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5566999999999998e-05 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144467+0000", 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8819999999999996e-06 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275371+0000", 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13090408000000001 2024-09-13T11:33:00.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275440+0000", 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8749000000000002e-05 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275530+0000", 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0236999999999997e-05 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282215+0000", 2024-09-13T11:33:00.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066849240000000001 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283575+0000", 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001360775 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283629+0000", 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283721+0000", 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1799999999999995e-05 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2326 4.16 4:6978b14d:::smithi20438004-23:head [write 2948352~396241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.980056598, 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13583619799999999, 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2326 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148113+0000", 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148305+0000", 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000192008 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148316+0000", 2024-09-13T11:33:00.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0293e-05 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148327+0000", 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1489e-05 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148398+0000", 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0421999999999998e-05 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148407+0000", 2024-09-13T11:33:00.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1090000000000004e-06 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279356+0000", 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13094884700000001 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279444+0000", 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:00.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279534+0000", 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9573000000000005e-05 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282581+0000", 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030473990000000001 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283856+0000", 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001275307 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283901+0000", 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5238999999999998e-05 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283947+0000", 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5972e-05 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2327 4.16 4:6978b14d:::smithi20438004-23:head [setxattr _header (56) in=7b,truncate 3344593] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:00.241 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.9798290979999997, 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13580746099999999, 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2327 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.34e-07 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148341+0000", 2024-09-13T11:33:00.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0999999999999998e-06 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148363+0000", 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2045999999999999e-05 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148369+0000", 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.665e-06 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148463+0000", 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4475999999999995e-05 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148474+0000", 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0319999999999999e-05 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280931+0000", 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132457142 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280994+0000", 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2693000000000004e-05 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281088+0000", 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3993000000000006e-05 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282632+0000", 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001544199 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283989+0000", 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013571670000000001 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284048+0000", 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9070999999999997e-05 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284146+0000", 2024-09-13T11:33:00.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8255e-05 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2328 4.16 4:6978b14d:::smithi20438004-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.9797945829999999, 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13592306100000001, 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:00.248 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2328 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148374+0000", 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6899999999999999e-07 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148375+0000", 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148378+0000", 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0989999999999999e-06 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148381+0000", 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148500+0000", 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119052 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:00.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148509+0000", 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8759999999999994e-06 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281377+0000", 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132868244 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281422+0000", 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4909e-05 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284171+0000", 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027497300000000001 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284216+0000", 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4626999999999998e-05 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284296+0000", 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0275e-05 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.253 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.254 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.254 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.254 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:00.256 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2151656 2024-09-13T11:33:00.256 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 856380 2024-09-13T11:33:00.256 INFO:tasks.rados.rados.0.smithi204.stdout:1165: finishing set_chunk to oid 49 2024-09-13T11:33:00.256 INFO:tasks.rados.rados.0.smithi204.stdout:1165: got expected ENOENT (src dne) 2024-09-13T11:33:00.256 INFO:tasks.rados.rados.0.smithi204.stdout:1167: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:00.256 INFO:tasks.rados.rados.0.smithi204.stdout:1166: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-13T11:33:00.256 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.256 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:00.254+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:00.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:00.276 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:33:00.279 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.351 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.355 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:00.354+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:00.356 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.R7NQA6cXzl is 19MB 2024-09-13T11:33:00.428 INFO:tasks.rados.rados.0.smithi204.stdout:1167: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:00.428 INFO:tasks.rados.rados.0.smithi204.stdout:1168: finishing set_chunk to oid 27 2024-09-13T11:33:00.428 INFO:tasks.rados.rados.0.smithi204.stdout:1168: got expected ENOENT (src dne) 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: oid 38 updating version 0 to 95 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: oid 38 updating version 95 to 98 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: oid 38 version 98 is already newer than 97 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: oid 38 version 98 is already newer than 96 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 98 (ObjNum 214 snap 0 seq_num 214) dirty exists 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1167: left oid 38 (ObjNum 214 snap 0 seq_num 214) 2024-09-13T11:33:00.429 INFO:tasks.rados.rados.0.smithi204.stdout:1170: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-13T11:33:00.446 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.473 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:00.473 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:00.473 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:00.473 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:00.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:00.478 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:00.455+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:00.519 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3293552 2024-09-13T11:33:00.519 INFO:tasks.rados.rados.0.smithi204.stdout:1169: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:33:00.520 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.555 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.595 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:00.560+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:00.636 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:00.636 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:00.636 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:00.636 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:00.636 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:00.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:00.641 INFO:tasks.rados.rados.0.smithi204.stdout:1171: finishing set_chunk to oid 4 2024-09-13T11:33:00.641 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 307712 length: 335360 2024-09-13T11:33:00.641 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:33:00.641 INFO:tasks.rados.rados.0.smithi204.stdout:1176: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-13T11:33:00.641 INFO:tasks.rados.rados.0.smithi204.stdout:1176: writing smithi20438004-33 from 413505 to 1049924 tid 1 2024-09-13T11:33:00.641 INFO:tasks.rados.rados.0.smithi204.stdout:1176: writing smithi20438004-33 from 1664805 to 2353869 tid 2 2024-09-13T11:33:00.651 INFO:tasks.rados.rados.0.smithi204.stdout:1176: writing smithi20438004-33 from 3038061 to 3530132 tid 3 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1145: done (15 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1151: done (14 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1159: done (13 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1160: done (12 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1165: done (11 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1166: done (10 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1167: done (9 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1168: done (8 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1169: done (7 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1170: done (6 left) 2024-09-13T11:33:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:1171: done (5 left) 2024-09-13T11:33:00.653 INFO:tasks.rados.rados.0.smithi204.stdout:1177: delete oid 47 current snap is 0 2024-09-13T11:33:00.653 INFO:tasks.rados.rados.0.smithi204.stdout:1173: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:33:00.653 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.682 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:00.662+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:00.724 INFO:tasks.rados.rados.0.smithi204.stdout:1172: finishing set_chunk to oid 46 2024-09-13T11:33:00.725 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 57 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-09-13T11:33:00.725 INFO:tasks.rados.rados.0.smithi204.stdout:1175: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:33:00.725 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.800 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:00.764+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:00.801 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1 4.e e38/31 4:76a15ccf:::smithi20438004-1:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.456418+0000", 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 34.328435380000002, 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046371939999999999, 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.802 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.456418+0000", 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.456418+0000", 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.456419+0000", 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5000000000000003e-07 2024-09-13T11:33:00.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457207+0000", 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078827599999999997 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457220+0000", 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3134e-05 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457230+0000", 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0084e-05 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457283+0000", 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2948000000000002e-05 2024-09-13T11:33:00.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.457302+0000", 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8675e-05 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.460991+0000", 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036891490000000001 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.461055+0000", 2024-09-13T11:33:00.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4377999999999999e-05 2024-09-13T11:33:00.808 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.808 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.808 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:73 4.f e38/31 4:f653a832:::smithi20438004-19:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:26.670905+0000", 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 34.113948076, 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004241224, 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670905+0000", 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670905+0000", 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.670906+0000", 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2399999999999999e-07 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671267+0000", 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00036090800000000001 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671278+0000", 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1188e-05 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671285+0000", 2024-09-13T11:33:00.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2119999999999997e-06 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671340+0000", 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4877999999999999e-05 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.671354+0000", 2024-09-13T11:33:00.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4095e-05 2024-09-13T11:33:00.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675095+0000", 2024-09-13T11:33:00.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037409019999999999 2024-09-13T11:33:00.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:26.675147+0000", 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1616999999999997e-05 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.815 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.763431558000001, 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005153202, 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021424+0000", 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8759999999999999e-06 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022585+0000", 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011614539999999999 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022596+0000", 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1110999999999999e-05 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022603+0000", 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4250000000000003e-06 2024-09-13T11:33:00.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022647+0000", 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4487999999999998e-05 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022662+0000", 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4817999999999999e-05 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.026542+0000", 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038797409999999999 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.026575+0000", 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3288999999999999e-05 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.820 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:189 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'10, mlcod=38'8)", 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.050082+0000", 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.734771438000003, 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049158680000000003, 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050082+0000", 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050082+0000", 2024-09-13T11:33:00.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050083+0000", 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9500000000000001e-07 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050921+0000", 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083774800000000003 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050932+0000", 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.172e-05 2024-09-13T11:33:00.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.050958+0000", 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5252000000000001e-05 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.051034+0000", 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6756999999999997e-05 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.051060+0000", 2024-09-13T11:33:00.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5868e-05 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.054961+0000", 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039008879999999999 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.054998+0000", 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6739999999999997e-05 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.390006657000001, 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078075499999999999, 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394848+0000", 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397492+0000", 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026441139999999999 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397513+0000", 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0381e-05 2024-09-13T11:33:00.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397520+0000", 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4789999999999999e-06 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397586+0000", 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5687999999999999e-05 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397612+0000", 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5870000000000001e-05 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402610+0000", 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049984199999999999 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402654+0000", 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4153000000000002e-05 2024-09-13T11:33:00.832 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "pg_update_log_missing(4.e epoch 38/31 rep_tid 32 entries 38'23 (0'0) error 4:7022fc62:::smithi20438004-6:head by client.4631.0:787 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 38'20 roll_forward_to 38'20)", 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.385279750000002, 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057797359999999997, 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8500000000000001e-07 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399583+0000", 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4139999999999994e-06 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399602+0000", 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9176000000000001e-05 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399611+0000", 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2520000000000005e-06 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399661+0000", 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9688000000000003e-05 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.405353+0000", 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056925209999999999 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.836 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1439 4.f e44/31 4:f5aacf61:::smithi20438004-27:head v 44'59, mlcod=43'57)", 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.972135087000002, 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039265439999999997, 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.812718+0000", 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6299999999999999e-07 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813007+0000", 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028868900000000001 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813018+0000", 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0575e-05 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813026+0000", 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2169999999999994e-06 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813096+0000", 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0528e-05 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.813115+0000", 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8508e-05 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.816613+0000", 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034979939999999999 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.816645+0000", 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1869999999999998e-05 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1440 4.f e44/31 4:f5aacf61:::smithi20438004-27:head v 44'60, mlcod=43'58)", 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:36.818815+0000", 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.96603829, 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004036291, 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.818815+0000", 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.818815+0000", 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.818816+0000", 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8400000000000003e-07 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819080+0000", 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026429999999999997 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819090+0000", 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3989999999999993e-06 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819099+0000", 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0939999999999997e-06 2024-09-13T11:33:00.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819211+0000", 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112755 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.819235+0000", 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3751e-05 2024-09-13T11:33:00.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.822823+0000", 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035873300000000001 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:36.822851+0000", 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8878e-05 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.753601565, 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081668139999999997, 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031253+0000", 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7900000000000007e-07 2024-09-13T11:33:00.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037708+0000", 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064553029999999999 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037720+0000", 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2062000000000001e-05 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037745+0000", 2024-09-13T11:33:00.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4766999999999999e-05 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037797+0000", 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2198999999999999e-05 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037815+0000", 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8178000000000001e-05 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039386+0000", 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015708110000000001 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039419+0000", 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2515000000000003e-05 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1715 4.f e51/50 4:fea8c5b2:::smithi20438004-39:head v 51'67, mlcod=0'0)", 2024-09-13T11:33:00.851 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.509084227000001, 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063741700000000002, 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275770+0000", 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9999999999999997e-07 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276160+0000", 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038996700000000002 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276171+0000", 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0702e-05 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276177+0000", 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7750000000000004e-06 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276230+0000", 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2751999999999999e-05 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276252+0000", 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2055000000000001e-05 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282092+0000", 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058396180000000004 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282143+0000", 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1601000000000003e-05 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1716 4.f e51/50 4:fea8c5b2:::smithi20438004-39:head v 51'68, mlcod=0'0)", 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.277466+0000", 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.507387344, 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050982550000000003, 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.857 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277466+0000", 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277466+0000", 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277467+0000", 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4799999999999997e-07 2024-09-13T11:33:00.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277545+0000", 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7830999999999994e-05 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277554+0000", 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.431e-06 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277560+0000", 2024-09-13T11:33:00.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5310000000000001e-06 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277601+0000", 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1674000000000001e-05 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277617+0000", 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5654e-05 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282535+0000", 2024-09-13T11:33:00.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049180550000000002 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282564+0000", 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9331000000000001e-05 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1717 4.f e51/50 4:fea8c5b2:::smithi20438004-39:head v 51'69, mlcod=0'0)", 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.278601+0000", 2024-09-13T11:33:00.861 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.506251927999999, 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039937180000000003, 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278601+0000", 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278601+0000", 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278602+0000", 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7299999999999995e-07 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278608+0000", 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7130000000000002e-06 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278618+0000", 2024-09-13T11:33:00.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0011e-05 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278625+0000", 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0620000000000003e-06 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.281299+0000", 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002673623 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.281315+0000", 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282567+0000", 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001251577 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282595+0000", 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8121999999999999e-05 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1752 4.9 e51/31 4:9441befd:::smithi20438004-8:head v 51'16, mlcod=41'14)", 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.394444+0000", 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.390409121999999, 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047770019999999998, 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.394444+0000", 2024-09-13T11:33:00.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.394444+0000", 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.394445+0000", 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7100000000000001e-07 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.395775+0000", 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013297509999999999 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.395785+0000", 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9070000000000002e-06 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.395808+0000", 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2997e-05 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.395851+0000", 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3618999999999997e-05 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.395871+0000", 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0081e-05 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.399188+0000", 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003317034 2024-09-13T11:33:00.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.399221+0000", 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2941999999999997e-05 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2037 4.e e54/51 4:76a15ccf:::smithi20438004-1:head v 54'47, mlcod=54'44)", 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:51.583520+0000", 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.2013337790000005, 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045556750000000003, 2024-09-13T11:33:00.870 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.583520+0000", 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.583520+0000", 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.583520+0000", 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.584095+0000", 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057482500000000005 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.584108+0000", 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3006e-05 2024-09-13T11:33:00.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.584114+0000", 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8819999999999998e-06 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.584194+0000", 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0377000000000006e-05 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.584209+0000", 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4398e-05 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.588038+0000", 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038292759999999999 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:51.588075+0000", 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7131000000000001e-05 2024-09-13T11:33:00.874 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2285 4.e e57/51 4:7984cf84:::smithi20438004-46:head v 57'55, mlcod=56'53)", 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5536190510000001, 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055185929999999996, 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231235+0000", 2024-09-13T11:33:00.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3799999999999997e-07 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232465+0000", 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001230182 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232475+0000", 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9650000000000003e-06 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232480+0000", 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1599999999999997e-06 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232538+0000", 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7711999999999999e-05 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232552+0000", 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.363e-05 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236731+0000", 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004179833 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236753+0000", 2024-09-13T11:33:00.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1472999999999999e-05 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2286 4.e e57/51 4:7984cf84:::smithi20438004-46:head v 57'56, mlcod=56'53)", 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:55.234602+0000", 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5502516799999997, 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049625650000000004, 2024-09-13T11:33:00.880 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234602+0000", 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234602+0000", 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234602+0000", 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1900000000000002e-07 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234760+0000", 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000158002 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234770+0000", 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2599999999999994e-06 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234774+0000", 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7400000000000004e-06 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234814+0000", 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9406999999999997e-05 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.234825+0000", 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1669e-05 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.239527+0000", 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047013849999999998 2024-09-13T11:33:00.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.239564+0000", 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7483e-05 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.885 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2287 4.e e57/51 4:7984cf84:::smithi20438004-46:head v 57'57, mlcod=56'53)", 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:55.235163+0000", 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5496899820000003, 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046210180000000002, 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.235163+0000", 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.235163+0000", 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.235164+0000", 2024-09-13T11:33:00.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.235168+0000", 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0509999999999998e-06 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.235175+0000", 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1500000000000002e-06 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.235179+0000", 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1500000000000001e-06 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236580+0000", 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014011150000000001 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236599+0000", 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8519e-05 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.239764+0000", 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031652210000000002 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.239784+0000", 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0265e-05 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2293 4.9 e57/56 4:9441befd:::smithi20438004-8:head v 57'22, mlcod=53'20)", 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:55.272897+0000", 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.5119561370000003, 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039781870000000002, 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.891 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.272897+0000", 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.272897+0000", 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.272898+0000", 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9500000000000002e-07 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.273698+0000", 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080020199999999997 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.273706+0000", 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1580000000000005e-06 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.273714+0000", 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9910000000000002e-06 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.273769+0000", 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5179000000000001e-05 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.273791+0000", 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1418999999999999e-05 2024-09-13T11:33:00.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.276200+0000", 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024090280000000001 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.276875+0000", 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00067551500000000001 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4b088bf8:::smithi20438004-24:head, version: 60'85, data_included: [1785856~905216], data_size: 905216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:4b088bf8:::smithi20438004-24:head@60'85, size: 3058874, copy_subset: [(0, 3058874)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3058874, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1785856, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:57.326288+0000", 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.4585652069999999, 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048531310000000001, 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.326288+0000", 2024-09-13T11:33:00.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.326288+0000", 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.326289+0000", 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2900000000000003e-07 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.330189+0000", 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039005310000000001 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.330207+0000", 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7983999999999999e-05 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.330226+0000", 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9014000000000001e-05 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.330315+0000", 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8647999999999996e-05 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.330330+0000", 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.468e-05 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.331141+0000", 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000811545 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.899 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4d4c8512:::smithi20438004-29:head, version: 55'82, data_included: [405504~978944,1437696~69632], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4d4c8512:::smithi20438004-29:head@55'82, size: 2806769, copy_subset: [(0, 2806769)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1507328, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.4507082370000002, 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064952710000000004, 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334146+0000", 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9500000000000002e-07 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339863+0000", 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057170859999999997 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339880+0000", 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6705999999999999e-05 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339897+0000", 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7663999999999999e-05 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339963+0000", 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5808000000000004e-05 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339972+0000", 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6289999999999999e-06 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.340640+0000", 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066878300000000005 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.903 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.904 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.904 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:00.904 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:00.904 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:00.907 INFO:tasks.rados.rados.0.smithi204.stdout:1174: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:33:00.907 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:00.907 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:00.906+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:00.918 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:00.958 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.R7NQA6cXzl 2024-09-13T11:33:00.979 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 522957 2024-09-13T11:33:01.025 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:01.025 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:01.025 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:01.026 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:01.041 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:01.047 INFO:tasks.rados.rados.0.smithi204.stdout:1176: finishing write tid 1 to smithi20438004-33 2024-09-13T11:33:01.047 INFO:tasks.rados.rados.0.smithi204.stdout:1176: finishing write tid 2 to smithi20438004-33 2024-09-13T11:33:01.047 INFO:tasks.rados.rados.0.smithi204.stdout:1176: finishing write tid 3 to smithi20438004-33 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: finishing write tid 4 to smithi20438004-33 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: finishing write tid 5 to smithi20438004-33 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: oid 33 updating version 0 to 64 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: oid 33 updating version 64 to 65 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: oid 33 version 65 is already newer than 62 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: oid 33 version 65 is already newer than 63 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 65 (ObjNum 215 snap 0 seq_num 215) dirty exists 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: left oid 33 (ObjNum 215 snap 0 seq_num 215) 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1172: done (5 left) 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1173: done (4 left) 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1174: done (3 left) 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1175: done (2 left) 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1176: done (1 left) 2024-09-13T11:33:01.048 INFO:tasks.rados.rados.0.smithi204.stdout:1177: done (0 left) 2024-09-13T11:33:01.049 INFO:tasks.rados.rados.0.smithi204.stdout:1178: delete oid 27 current snap is 0 2024-09-13T11:33:01.049 INFO:tasks.rados.rados.0.smithi204.stdout:1178: done (0 left) 2024-09-13T11:33:01.049 INFO:tasks.rados.rados.0.smithi204.stdout:1179: read oid 2 snap -1 2024-09-13T11:33:01.049 INFO:tasks.rados.rados.0.smithi204.stdout:1179: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-13T11:33:01.049 INFO:tasks.rados.rados.0.smithi204.stdout:1180: read oid 1 snap -1 2024-09-13T11:33:01.049 INFO:tasks.rados.rados.0.smithi204.stdout:1180: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-13T11:33:01.049 INFO:tasks.rados.rados.0.smithi204.stdout:1181: set_chunk oid 4 target oid 2024-09-13T11:33:01.053 INFO:tasks.rados.rados.0.smithi204.stdout:1181: set_chunk oid 4 offset: 240128 length: 824320 target oid 64ee3569f593d6229913262493b27eec6484e5fcf094bd229d903d71a0ce38d5 offset: 0 snap -1 2024-09-13T11:33:01.053 INFO:tasks.rados.rados.0.smithi204.stdout:1182: set_chunk oid 45 target oid 2024-09-13T11:33:01.057 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:01.023+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:01.062 INFO:tasks.rados.rados.0.smithi204.stdout:1182: set_chunk oid 45 offset: 1879552 length: 1427456 target oid cf1a86492ac6def97884e031652bc8b4172191d324a87d8ca1b0cc466f59c507 offset: 0 snap -1 2024-09-13T11:33:01.062 INFO:tasks.rados.rados.0.smithi204.stdout:1183: read oid 13 snap -1 2024-09-13T11:33:01.062 INFO:tasks.rados.rados.0.smithi204.stdout:1183: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:01.062 INFO:tasks.rados.rados.0.smithi204.stdout:1184: write oid 29 current snap is 0 2024-09-13T11:33:01.062 INFO:tasks.rados.rados.0.smithi204.stdout:1184: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1184: writing smithi20438004-29 from 463926 to 926092 tid 1 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1184: writing smithi20438004-29 from 1723898 to 2160289 tid 2 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1184: writing smithi20438004-29 from 2603977 to 3329864 tid 3 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1185: read oid 15 snap -1 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1185: expect deleted 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1186: read oid 46 snap -1 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1186: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:01.091 INFO:tasks.rados.rados.0.smithi204.stdout:1187: write oid 22 current snap is 0 2024-09-13T11:33:01.092 INFO:tasks.rados.rados.0.smithi204.stdout:1187: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-13T11:33:01.092 INFO:tasks.rados.rados.0.smithi204.stdout:1187: writing smithi20438004-22 from 480733 to 1018148 tid 1 2024-09-13T11:33:01.097 INFO:tasks.rados.rados.0.smithi204.stdout:1187: writing smithi20438004-22 from 1782380 to 2478782 tid 2 2024-09-13T11:33:01.110 INFO:tasks.rados.rados.0.smithi204.stdout:1187: writing smithi20438004-22 from 3031400 to 3568559 tid 3 2024-09-13T11:33:01.110 INFO:tasks.rados.rados.0.smithi204.stdout:1188: write (excl) oid 8 current snap is 0 2024-09-13T11:33:01.110 INFO:tasks.rados.rados.0.smithi204.stdout:1188: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-13T11:33:01.110 INFO:tasks.rados.rados.0.smithi204.stdout:1188: writing smithi20438004-8 from 497540 to 1110204 tid 1 2024-09-13T11:33:01.115 INFO:tasks.rados.rados.0.smithi204.stdout:1188: writing smithi20438004-8 from 1840862 to 2280922 tid 2 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1188: writing smithi20438004-8 from 3058823 to 3523607 tid 3 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1189: read oid 12 snap -1 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1189: expect (ObjNum 194 snap 0 seq_num 194) 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1190: tier_promote oid 24 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1191: read oid 28 snap -1 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1191: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1192: write oid 21 current snap is 0 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1192: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-13T11:33:01.149 INFO:tasks.rados.rados.0.smithi204.stdout:1192: writing smithi20438004-21 from 514347 to 918613 tid 1 2024-09-13T11:33:01.150 INFO:tasks.rados.rados.0.smithi204.stdout:1192: writing smithi20438004-21 from 1499344 to 2199415 tid 2 2024-09-13T11:33:01.150 INFO:tasks.rados.rados.0.smithi204.stdout:1192: writing smithi20438004-21 from 2686246 to 3362302 tid 3 2024-09-13T11:33:01.150 INFO:tasks.rados.rados.0.smithi204.stdout:1193: tier_evict oid 27 2024-09-13T11:33:01.150 INFO:tasks.rados.rados.0.smithi204.stdout:1193: tier_evict oid 27 snap -1 2024-09-13T11:33:01.150 INFO:tasks.rados.rados.0.smithi204.stdout:1180: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-13T11:33:01.153 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:01.143+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:01.174 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:01.175 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:01.175 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:01.213 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:01.240 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 176559 2024-09-13T11:33:01.280 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:01.266+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:01.296 INFO:tasks.rados.rados.0.smithi204.stdout:1179: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-13T11:33:01.350 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.990623937999999, 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.351 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:33:01.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.356 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.242093017000002, 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:33:01.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.229875423000003, 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:33:01.361 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:01.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:33:01.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:33:01.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:33:01.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:33:01.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:33:01.366 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1679 4.4 4:2b094039:::smithi20438004-40:head [tier-flush] snapc 0=[] RETRY=3 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 16.075751316000002, 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0158098280000001, 2024-09-13T11:33:01.367 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1679 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:01.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269187+0000", 2024-09-13T11:33:01.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9479999999999998e-06 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269193+0000", 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.173e-06 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269209+0000", 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6039e-05 2024-09-13T11:33:01.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269224+0000", 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5494000000000001e-05 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270767+0000", 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015428869999999999 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270782+0000", 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.519e-05 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284573+0000", 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013791077000000001 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284717+0000", 2024-09-13T11:33:01.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014369599999999999 2024-09-13T11:33:01.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284994+0000", 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.000277324 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.374 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2310 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:01.375 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:01.375 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.2451297710000002, 2024-09-13T11:33:01.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17044168200000001, 2024-09-13T11:33:01.375 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.375 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.375 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2310 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:01.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:01.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099808+0000", 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100126+0000", 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031826000000000001 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100144+0000", 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8029999999999998e-05 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100153+0000", 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0119999999999994e-06 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100230+0000", 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6221000000000001e-05 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100242+0000", 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.257e-05 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270222+0000", 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16997934100000001 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270248+0000", 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6366000000000001e-05 2024-09-13T11:33:01.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2311 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.2385894950000003, 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16403588, 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2311 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106347+0000", 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106673+0000", 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588799999999998 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106687+0000", 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3247e-05 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106693+0000", 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2249999999999997e-06 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106741+0000", 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8715999999999999e-05 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106750+0000", 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3159999999999997e-06 2024-09-13T11:33:01.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270375+0000", 2024-09-13T11:33:01.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16362501300000001 2024-09-13T11:33:01.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270382+0000", 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.386 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2312 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.2332982330000002, 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158778577, 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2312 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111639+0000", 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:33:01.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111756+0000", 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117166 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111770+0000", 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3492999999999999e-05 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111778+0000", 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5820000000000004e-06 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111834+0000", 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6564000000000003e-05 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111842+0000", 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8280000000000003e-06 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270410+0000", 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15856827400000001 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270416+0000", 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.391 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2313 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.2331439020000001, 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158679564, 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2313 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6399999999999998e-07 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111795+0000", 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0450000000000001e-06 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111799+0000", 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5250000000000001e-06 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111802+0000", 2024-09-13T11:33:01.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.162e-06 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111876+0000", 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4062999999999999e-05 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111885+0000", 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0850000000000003e-06 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270465+0000", 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158579839 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270472+0000", 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4810000000000003e-06 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4.8a282f38 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:01.396 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.2331136860000003, 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15868154300000001, 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.397 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:01.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111823+0000", 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3299999999999998e-07 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111824+0000", 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7290000000000001e-06 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111829+0000", 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5859999999999998e-06 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111832+0000", 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.456e-06 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111901+0000", 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8447000000000004e-05 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111905+0000", 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8569999999999997e-06 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270497+0000", 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158592381 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270504+0000", 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.754e-06 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2309 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.0756185680000003, 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.00703426, 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2309 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269319+0000", 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.265e-06 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269324+0000", 2024-09-13T11:33:01.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1800000000000004e-06 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269334+0000", 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7310000000000005e-06 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269343+0000", 2024-09-13T11:33:01.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8339999999999994e-06 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269381+0000", 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7709000000000001e-05 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269391+0000", 2024-09-13T11:33:01.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:01.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278002+0000", 2024-09-13T11:33:01.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086114859999999998 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278150+0000", 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014796199999999999 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.276352+0000", 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99820153300000003 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2170 4.12 4:4b088bf8:::smithi20438004-24:head [write 2083958~603055] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:01.410 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0670527060000001, 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029083859, 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2170 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:01.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277884+0000", 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4399999999999998e-07 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278124+0000", 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000239485 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278133+0000", 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3219999999999999e-06 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278141+0000", 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:33:01.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280144+0000", 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020033289999999999 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280155+0000", 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0966e-05 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301300+0000", 2024-09-13T11:33:01.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021145557999999998 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301414+0000", 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011306099999999999 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301601+0000", 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000187357 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304518+0000", 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029169539999999998 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304561+0000", 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229000000000002e-05 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306715+0000", 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002153706 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306792+0000", 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7466999999999995e-05 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306851+0000", 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8811999999999999e-05 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306967+0000", 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116062 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.417 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2171 4.12 4:4b088bf8:::smithi20438004-24:head [setxattr _header (56) in=7b,truncate 3058874] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0667801380000004, 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029067939000000001, 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.418 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2171 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278157+0000", 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0699999999999996e-07 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278161+0000", 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2640000000000002e-06 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278173+0000", 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1868999999999999e-05 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278179+0000", 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280537+0000", 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023576320000000001 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280544+0000", 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7560000000000002e-06 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303651+0000", 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023107240000000001 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303802+0000", 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150774 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303982+0000", 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018009800000000001 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306699+0000", 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027165100000000001 2024-09-13T11:33:01.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306720+0000", 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1535e-05 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306753+0000", 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3019999999999999e-05 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307010+0000", 2024-09-13T11:33:01.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025724800000000001 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307061+0000", 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0633000000000001e-05 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307224+0000", 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000162843 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2172 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0667474539999997, 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029398245999999999, 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.426 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2172 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:01.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000005e-07 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278193+0000", 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278201+0000", 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278207+0000", 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4289999999999998e-06 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280568+0000", 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023610900000000002 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280577+0000", 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8259999999999999e-06 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304605+0000", 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024027829000000001 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304704+0000", 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9610000000000003e-05 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307334+0000", 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002629667 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307421+0000", 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7433000000000001e-05 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307587+0000", 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016528800000000001 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2219 4.12 4:4d4c8512:::smithi20438004-29:head [sparse-read 0~2348017] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0666762460000001, 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029874108999999999, 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2219 2024-09-13T11:33:01.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3599999999999999e-07 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278262+0000", 2024-09-13T11:33:01.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2110000000000001e-06 2024-09-13T11:33:01.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278269+0000", 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.314e-06 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278274+0000", 2024-09-13T11:33:01.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5249999999999996e-06 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280850+0000", 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025761389999999999 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280858+0000", 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8760000000000005e-06 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307628+0000", 2024-09-13T11:33:01.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.026769465999999999 2024-09-13T11:33:01.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307703+0000", 2024-09-13T11:33:01.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4847999999999999e-05 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.308134+0000", 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00043129399999999998 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.440 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0649207790000004, 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029229465, 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.441 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280016+0000", 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280020+0000", 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280026+0000", 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1070000000000001e-06 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280032+0000", 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280109+0000", 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6298999999999996e-05 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280116+0000", 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5129999999999999e-06 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298271+0000", 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018154805 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298334+0000", 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3268999999999993e-05 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309030+0000", 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010695566 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309089+0000", 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309245+0000", 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000155317 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2340 4.18 4:1d367c64:::smithi20438004-45:head [write 578207~550227] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0648930319999996, 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030475704999999999, 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.448 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2340 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0399999999999996e-07 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280452+0000", 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040812800000000001 2024-09-13T11:33:01.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280461+0000", 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3829999999999998e-06 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280468+0000", 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0400000000000004e-06 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280506+0000", 2024-09-13T11:33:01.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8037000000000002e-05 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280514+0000", 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.011e-06 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298673+0000", 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018159411 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298829+0000", 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015594 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.299023+0000", 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193838 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307549+0000", 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085256289999999998 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307635+0000", 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6596999999999998e-05 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307689+0000", 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4027999999999998e-05 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307748+0000", 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8733999999999997e-05 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310255+0000", 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025070940000000001 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310375+0000", 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119691 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310519+0000", 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014364 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.455 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2341 4.18 4:1d367c64:::smithi20438004-45:head [write 1762410~779734] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0644506649999999, 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031687067999999999, 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.456 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2341 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280486+0000", 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281018+0000", 2024-09-13T11:33:01.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053231899999999996 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281027+0000", 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.704e-06 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281034+0000", 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0659999999999998e-06 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281159+0000", 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124661 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281168+0000", 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000001e-06 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302725+0000", 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021557478000000001 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302889+0000", 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016421299999999999 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303087+0000", 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019817200000000001 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307806+0000", 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004718168 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307868+0000", 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2775999999999997e-05 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307920+0000", 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1749000000000002e-05 2024-09-13T11:33:01.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307971+0000", 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1343e-05 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311829+0000", 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038571769999999998 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312043+0000", 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021412199999999999 2024-09-13T11:33:01.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312172+0000", 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012966900000000001 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.464 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2342 4.18 4:1d367c64:::smithi20438004-45:head [write 3286845~91362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0638835139999996, 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031505618999999999, 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2342 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281053+0000", 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2799999999999995e-07 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281104+0000", 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0550999999999998e-05 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281112+0000", 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0129999999999993e-06 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281118+0000", 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281191+0000", 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2738999999999996e-05 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281197+0000", 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309351+0000", 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028153293999999999 2024-09-13T11:33:01.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309441+0000", 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0048000000000002e-05 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309594+0000", 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153609 2024-09-13T11:33:01.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311315+0000", 2024-09-13T11:33:01.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017209829999999999 2024-09-13T11:33:01.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311795+0000", 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048009300000000001 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312281+0000", 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048612699999999999 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312394+0000", 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112935 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312404+0000", 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0112000000000001e-05 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312449+0000", 2024-09-13T11:33:01.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4650000000000001e-05 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312558+0000", 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108959 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2343 4.18 4:1d367c64:::smithi20438004-45:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.474 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0638079640000004, 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031723945000000003, 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2343 2024-09-13T11:33:01.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281129+0000", 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281132+0000", 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.988e-06 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281137+0000", 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.95e-06 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281142+0000", 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4e-06 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281219+0000", 2024-09-13T11:33:01.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6946000000000005e-05 2024-09-13T11:33:01.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281225+0000", 2024-09-13T11:33:01.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.601e-06 2024-09-13T11:33:01.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310552+0000", 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029326497 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310643+0000", 2024-09-13T11:33:01.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1416999999999996e-05 2024-09-13T11:33:01.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:01.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310817+0000", 2024-09-13T11:33:01.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000173463 2024-09-13T11:33:01.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312229+0000", 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001412424 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312324+0000", 2024-09-13T11:33:01.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4264000000000004e-05 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312578+0000", 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025401699999999999 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312605+0000", 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7557000000000001e-05 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312650+0000", 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4496000000000002e-05 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312683+0000", 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2898000000000002e-05 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312852+0000", 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000169436 2024-09-13T11:33:01.485 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2344 4.18 4:1d367c64:::smithi20438004-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.0637882169999999, 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032021714999999999, 2024-09-13T11:33:01.486 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2344 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:01.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.15e-07 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281150+0000", 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.857e-06 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281155+0000", 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0579999999999997e-06 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281160+0000", 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0409999999999997e-06 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281248+0000", 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7979000000000006e-05 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281255+0000", 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5089999999999999e-06 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311169+0000", 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02991394 2024-09-13T11:33:01.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311266+0000", 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7007000000000001e-05 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312894+0000", 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016280489999999999 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312976+0000", 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2052999999999995e-05 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.313169+0000", 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193807 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.493 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:01.497 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:01.498 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2595615 2024-09-13T11:33:01.498 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:01.498 INFO:tasks.rados.rados.0.smithi204.stdout:1181: finishing set_chunk to oid 4 2024-09-13T11:33:01.498 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 240128 length: 824320 2024-09-13T11:33:01.498 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-13T11:33:01.498 INFO:tasks.rados.rados.0.smithi204.stdout:1182: finishing set_chunk to oid 45 2024-09-13T11:33:01.499 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 56 (ObjNum 201 snap 0 seq_num 201) dirty exists 2024-09-13T11:33:01.499 INFO:tasks.rados.rados.0.smithi204.stdout:1184: finishing write tid 1 to smithi20438004-29 2024-09-13T11:33:01.499 INFO:tasks.rados.rados.0.smithi204.stdout:1183: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:01.502 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:01.495+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:01.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-12 --journal-path /var/lib/ceph/osd/ceph-12/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.3.2 2024-09-13T11:33:01.557 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 950487 2024-09-13T11:33:01.597 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:01.596+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:01.642 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:01.642 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:01.642 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:01.642 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:01.651 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:01.698 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:01.696+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1184: finishing write tid 2 to smithi20438004-29 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1179: done (14 left) 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1180: done (13 left) 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1181: done (12 left) 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1182: done (11 left) 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1183: done (10 left) 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1194: tier_evict oid 39 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1194: tier_evict oid 39 snap -1 2024-09-13T11:33:01.699 INFO:tasks.rados.rados.0.smithi204.stdout:1184: finishing write tid 3 to smithi20438004-29 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1184: finishing write tid 4 to smithi20438004-29 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1184: finishing write tid 5 to smithi20438004-29 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1184: oid 29 updating version 0 to 87 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1184: oid 29 updating version 87 to 88 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1184: oid 29 version 88 is already newer than 86 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1184: oid 29 updating version 88 to 89 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 89 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1184: left oid 29 (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:01.700 INFO:tasks.rados.rados.0.smithi204.stdout:1186: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:01.711 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1393784 2024-09-13T11:33:01.793 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:01.793 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:01.793 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:01.793 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:01.793 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:01.798 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:01.797+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:01.801 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:01.877 INFO:tasks.rados.rados.0.smithi204.stdout:1187: finishing write tid 1 to smithi20438004-22 2024-09-13T11:33:01.877 INFO:tasks.rados.rados.0.smithi204.stdout:1187: finishing write tid 2 to smithi20438004-22 2024-09-13T11:33:01.877 INFO:tasks.rados.rados.0.smithi204.stdout:1187: finishing write tid 3 to smithi20438004-22 2024-09-13T11:33:01.877 INFO:tasks.rados.rados.0.smithi204.stdout:1187: finishing write tid 4 to smithi20438004-22 2024-09-13T11:33:01.877 INFO:tasks.rados.rados.0.smithi204.stdout:1184: done (10 left) 2024-09-13T11:33:01.877 INFO:tasks.rados.rados.0.smithi204.stdout:1185: done (9 left) 2024-09-13T11:33:01.877 INFO:tasks.rados.rados.0.smithi204.stdout:1186: done (8 left) 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1195: read oid 41 snap -1 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1195: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1196: read oid 49 snap -1 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1196: expect deleted 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1197: write oid 36 current snap is 0 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1187: finishing write tid 5 to smithi20438004-22 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1187: oid 22 updating version 0 to 77 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1187: oid 22 version 77 is already newer than 75 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1187: oid 22 updating version 77 to 78 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1187: oid 22 version 78 is already newer than 76 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 78 (ObjNum 219 snap 0 seq_num 219) dirty exists 2024-09-13T11:33:01.878 INFO:tasks.rados.rados.0.smithi204.stdout:1187: left oid 22 (ObjNum 219 snap 0 seq_num 219) 2024-09-13T11:33:01.879 INFO:tasks.rados.rados.0.smithi204.stdout:1188: finishing write tid 1 to smithi20438004-8 2024-09-13T11:33:01.879 INFO:tasks.rados.rados.0.smithi204.stdout:1197: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-13T11:33:01.882 INFO:tasks.rados.rados.0.smithi204.stdout:1197: writing smithi20438004-36 from 531154 to 1010669 tid 1 2024-09-13T11:33:01.887 INFO:tasks.rados.rados.0.smithi204.stdout:1197: writing smithi20438004-36 from 1557826 to 2001555 tid 2 2024-09-13T11:33:01.894 INFO:tasks.rados.rados.0.smithi204.stdout:1197: writing smithi20438004-36 from 2713669 to 3200997 tid 3 2024-09-13T11:33:01.897 INFO:tasks.rados.rados.0.smithi204.stdout:1197: writing smithi20438004-36 from 3679295 to 3731154 tid 4 2024-09-13T11:33:01.897 INFO:tasks.rados.rados.0.smithi204.stdout:1187: done (10 left) 2024-09-13T11:33:01.898 INFO:tasks.rados.rados.0.smithi204.stdout:1198: read oid 19 snap -1 2024-09-13T11:33:01.898 INFO:tasks.rados.rados.0.smithi204.stdout:1198: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:33:01.898 INFO:tasks.rados.rados.0.smithi204.stdout:1199: write oid 44 current snap is 0 2024-09-13T11:33:01.898 INFO:tasks.rados.rados.0.smithi204.stdout:1188: finishing write tid 2 to smithi20438004-8 2024-09-13T11:33:01.898 INFO:tasks.rados.rados.0.smithi204.stdout:1189: expect (ObjNum 194 snap 0 seq_num 194) 2024-09-13T11:33:01.899 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:01.898+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:01.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:01.950 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:01.950 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:01.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:01.950 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:01.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.950 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.850776003999997, 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:33:01.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.955 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.838341859000003, 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:33:01.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.828388265000001, 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:33:01.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:33:01.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:33:01.965 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.797670457999999, 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:33:01.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.778847139, 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:33:01.970 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.975 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.765327790000001, 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:01.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:33:01.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.980 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.755687457000001, 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:33:01.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.745381956999999, 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:33:01.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:33:01.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:33:01.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.734996015, 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:33:01.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:36 3.3 3:e2816a99:::d4bb749789c90c884ffd20ac62f804c7a6b80502d14432fa627e1239249ee2da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:01.995 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.677468616000001, 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013305613000000001, 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270480+0000", 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2399999999999998e-07 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270655+0000", 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017481300000000001 2024-09-13T11:33:01.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270683+0000", 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8340999999999999e-05 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270693+0000", 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-09-13T11:33:01.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271029+0000", 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033640200000000002 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271038+0000", 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278355+0000", 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073166510000000004 2024-09-13T11:33:01.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278481+0000", 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125955 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278657+0000", 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176816 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280615+0000", 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019576810000000002 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283538+0000", 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029229030000000001 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283592+0000", 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4366000000000001e-05 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:02.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283671+0000", 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8337000000000006e-05 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283785+0000", 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114412 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:40 3.3 3:c7a12847:::768a08ae37e9bf19369a517906ebd209c1f2b8ad3684596d644da7d48b42114d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:02.002 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.67722551, 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013347114, 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:02.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270839+0000", 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115369 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270847+0000", 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270854+0000", 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271073+0000", 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021964400000000001 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271080+0000", 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7909999999999999e-06 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278928+0000", 2024-09-13T11:33:02.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078479819999999999 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279058+0000", 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130102 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279300+0000", 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024166300000000001 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280641+0000", 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013409229999999999 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283545+0000", 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002904401 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283927+0000", 2024-09-13T11:33:02.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038193199999999997 2024-09-13T11:33:02.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:02.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283964+0000", 2024-09-13T11:33:02.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7117999999999999e-05 2024-09-13T11:33:02.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284070+0000", 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010545899999999999 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:41 3.3 3:e9fa9a15:::3c65c6c9770b7a29cd5a9b6dc5b0b39ab23cb2a64bfb19442efcb911d255e067:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.677081897000001, 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013499547000000001, 2024-09-13T11:33:02.010 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270867+0000", 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270893+0000", 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6057000000000001e-05 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270901+0000", 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:33:02.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270915+0000", 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4450000000000001e-05 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271104+0000", 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018904600000000001 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271111+0000", 2024-09-13T11:33:02.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6039999999999998e-06 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280674+0000", 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095629140000000005 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280811+0000", 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136802 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:02.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281027+0000", 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000216617 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283802+0000", 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027746540000000001 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283884+0000", 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2167000000000001e-05 2024-09-13T11:33:02.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284178+0000", 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029347899999999997 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284246+0000", 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8609999999999995e-05 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284366+0000", 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011950799999999999 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:43 3.3 3:dc8816b1:::1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.677021202000001, 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013665096999999999, 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:02.017 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-13T11:33:02.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270933+0000", 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2410000000000003e-06 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270939+0000", 2024-09-13T11:33:02.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5810000000000002e-06 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270945+0000", 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271134+0000", 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000189161 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271140+0000", 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2999999999999998e-06 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281407+0000", 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010266226 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281549+0000", 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142583 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281737+0000", 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018791500000000001 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283810+0000", 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002072571 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284127+0000", 2024-09-13T11:33:02.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000317516 2024-09-13T11:33:02.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284417+0000", 2024-09-13T11:33:02.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029000899999999998 2024-09-13T11:33:02.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284480+0000", 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3136e-05 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284592+0000", 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111712 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.10.22:39 3.3 3:c55f8587:::a2f2c330500f1b1d28c41e464817565848095218a89d32db0b731691bbe2b77b:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.654825432999999, 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011011337, 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:33:02.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:33:02.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293124+0000", 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5319999999999999e-06 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293132+0000", 2024-09-13T11:33:02.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293143+0000", 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1477999999999999e-05 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293152+0000", 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8289999999999997e-06 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298197+0000", 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050456870000000001 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298273+0000", 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5799999999999999e-05 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298497+0000", 2024-09-13T11:33:02.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000223618 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.299173+0000", 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00067642199999999998 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.299376+0000", 2024-09-13T11:33:02.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020231700000000001 2024-09-13T11:33:02.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:02.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304021+0000", 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004645003 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304068+0000", 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7139999999999999e-05 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304134+0000", 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6203000000000003e-05 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.1031933299999999, 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43316489499999999, 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.033 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844756+0000", 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844760+0000", 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0550000000000001e-06 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844769+0000", 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6129999999999993e-06 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844776+0000", 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0280000000000003e-06 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:02.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7639000000000002e-05 2024-09-13T11:33:02.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844841+0000", 2024-09-13T11:33:02.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1049999999999997e-06 2024-09-13T11:33:02.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277896+0000", 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43305493499999997 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277920+0000", 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3614000000000001e-05 2024-09-13T11:33:02.039 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.1031416969999999, 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43315226600000001, 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:02.040 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844807+0000", 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3200000000000003e-07 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844811+0000", 2024-09-13T11:33:02.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.799e-06 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844827+0000", 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5461999999999999e-05 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.126e-06 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5369999999999999e-05 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844863+0000", 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.658e-06 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.044 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277951+0000", 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308847900000003 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277959+0000", 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:02.045 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 6.1030999860000001, 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43314228399999999, 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.046 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844849+0000", 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0100000000000005e-07 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844853+0000", 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9589999999999997e-06 2024-09-13T11:33:02.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:02.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4110000000000001e-06 2024-09-13T11:33:02.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844864+0000", 2024-09-13T11:33:02.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8930000000000004e-06 2024-09-13T11:33:02.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844892+0000", 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8042e-05 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844898+0000", 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5049999999999998e-06 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277984+0000", 2024-09-13T11:33:02.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308647700000003 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277990+0000", 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3960000000000003e-06 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4:8c30e60e:::smithi20438004-18:head [read 0~2487436,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.6655506639999995, 2024-09-13T11:33:02.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012791258999999999, 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:02.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282399+0000", 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.691e-06 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282404+0000", 2024-09-13T11:33:02.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.887e-06 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282414+0000", 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9539999999999999e-06 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282423+0000", 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6039999999999993e-06 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282477+0000", 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4738999999999999e-05 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282486+0000", 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5450000000000003e-06 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287493+0000", 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050068949999999999 2024-09-13T11:33:02.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287636+0000", 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014333299999999999 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.295189+0000", 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075526109999999999 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 153896~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.6654490739999996, 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014013433, 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:02.058 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282500+0000", 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5499999999999998e-07 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282503+0000", 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.563e-06 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282510+0000", 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8820000000000003e-06 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282516+0000", 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282562+0000", 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5868999999999997e-05 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282570+0000", 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296085+0000", 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013515255 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296187+0000", 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101457 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296512+0000", 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588099999999998 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 447518~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.6653645810000004, 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014496826000000001, 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:02.063 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5000000000000002e-07 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:02.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282589+0000", 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4220000000000002e-06 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282595+0000", 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8789999999999997e-06 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282606+0000", 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0229e-05 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282652+0000", 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6610999999999998e-05 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282661+0000", 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2640000000000008e-06 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296550+0000", 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013888981999999999 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296628+0000", 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8564000000000002e-05 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.297080+0000", 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000452225 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:02.068 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:02.069 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:02.070 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:02.070 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:02.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:02.071 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:02.072 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: finishing write tid 3 to smithi20438004-8 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: finishing write tid 4 to smithi20438004-8 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: finishing write tid 5 to smithi20438004-8 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: oid 8 updating version 0 to 25 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: oid 8 updating version 25 to 28 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: oid 8 version 28 is already newer than 27 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: oid 8 version 28 is already newer than 26 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 28 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1188: left oid 8 (ObjNum 220 snap 0 seq_num 220) 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:1190: got (0) Success 2024-09-13T11:33:02.072 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 85 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-09-13T11:33:02.073 INFO:tasks.rados.rados.0.smithi204.stdout:1192: finishing write tid 1 to smithi20438004-21 2024-09-13T11:33:02.073 INFO:tasks.rados.rados.0.smithi204.stdout:1191: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-13T11:33:02.073 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:02.068+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:02.082 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:02.082 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:02.098 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:02.098 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:02.100 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:02.123 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:02.123 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:02.140 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:02.142 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:02.155 INFO:tasks.rados.rados.0.smithi204.stdout:1192: finishing write tid 2 to smithi20438004-21 2024-09-13T11:33:02.155 INFO:tasks.rados.rados.0.smithi204.stdout:1193: got (22) Invalid argument 2024-09-13T11:33:02.155 INFO:tasks.rados.rados.0.smithi204.stdout:1192: finishing write tid 3 to smithi20438004-21 2024-09-13T11:33:02.155 INFO:tasks.rados.rados.0.smithi204.stdout:1199: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-13T11:33:02.158 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:02.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:02.161 INFO:tasks.rados.rados.0.smithi204.stdout:1199: writing smithi20438004-44 from 547961 to 1102725 tid 1 2024-09-13T11:33:02.168 INFO:tasks.rados.rados.0.smithi204.stdout:1199: writing smithi20438004-44 from 1616308 to 2320048 tid 2 2024-09-13T11:33:02.170 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:02.169+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:02.175 INFO:tasks.rados.rados.0.smithi204.stdout:1199: writing smithi20438004-44 from 2741092 to 3439692 tid 3 2024-09-13T11:33:02.175 INFO:tasks.rados.rados.0.smithi204.stdout:1188: done (11 left) 2024-09-13T11:33:02.175 INFO:tasks.rados.rados.0.smithi204.stdout:1189: done (10 left) 2024-09-13T11:33:02.175 INFO:tasks.rados.rados.0.smithi204.stdout:1190: done (9 left) 2024-09-13T11:33:02.175 INFO:tasks.rados.rados.0.smithi204.stdout:1191: done (8 left) 2024-09-13T11:33:02.175 INFO:tasks.rados.rados.0.smithi204.stdout:1193: done (7 left) 2024-09-13T11:33:02.176 INFO:tasks.rados.rados.0.smithi204.stdout:1200: read oid 32 snap -1 2024-09-13T11:33:02.176 INFO:tasks.rados.rados.0.smithi204.stdout:1200: expect (ObjNum 202 snap 0 seq_num 202) 2024-09-13T11:33:02.176 INFO:tasks.rados.rados.0.smithi204.stdout:1201: read oid 34 snap -1 2024-09-13T11:33:02.176 INFO:tasks.rados.rados.0.smithi204.stdout:1201: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:02.176 INFO:tasks.rados.rados.0.smithi204.stdout:1202: set_chunk oid 26 target oid 2024-09-13T11:33:02.177 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:02.177 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:02.194 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:02.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:02.206 INFO:tasks.rados.rados.0.smithi204.stdout:1202: set_chunk oid 26 offset: 363008 length: 2507776 target oid 5d6daf3d044e92f17444d68009b05157d3ec62c41179a6534f8c9d2bd159fb39 offset: 0 snap -1 2024-09-13T11:33:02.206 INFO:tasks.rados.rados.0.smithi204.stdout:1203: delete oid 10 current snap is 0 2024-09-13T11:33:02.206 INFO:tasks.rados.rados.0.smithi204.stdout:1192: finishing write tid 4 to smithi20438004-21 2024-09-13T11:33:02.206 INFO:tasks.rados.rados.0.smithi204.stdout:1192: finishing write tid 5 to smithi20438004-21 2024-09-13T11:33:02.206 INFO:tasks.rados.rados.0.smithi204.stdout:1192: oid 21 updating version 0 to 26 2024-09-13T11:33:02.206 INFO:tasks.rados.rados.0.smithi204.stdout:1192: oid 21 version 26 is already newer than 25 2024-09-13T11:33:02.207 INFO:tasks.rados.rados.0.smithi204.stdout:1192: oid 21 version 26 is already newer than 24 2024-09-13T11:33:02.207 INFO:tasks.rados.rados.0.smithi204.stdout:1192: oid 21 updating version 26 to 27 2024-09-13T11:33:02.207 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 27 (ObjNum 221 snap 0 seq_num 221) dirty exists 2024-09-13T11:33:02.207 INFO:tasks.rados.rados.0.smithi204.stdout:1192: left oid 21 (ObjNum 221 snap 0 seq_num 221) 2024-09-13T11:33:02.207 INFO:tasks.rados.rados.0.smithi204.stdout:1194: got (22) Invalid argument 2024-09-13T11:33:02.207 INFO:tasks.rados.rados.0.smithi204.stdout:1195: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:33:02.211 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:02.212 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:02.213 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:02.214 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:02.214 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:02.214 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:02.223 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:02.228 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:02.228 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:02.228 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:02.246 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:02.247 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:02.263 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:02.263 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:02.271 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:02.269+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:02.279 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:02.280 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:02.283 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:02.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:33:02.292 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1417404 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: finishing write tid 1 to smithi20438004-36 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: finishing write tid 2 to smithi20438004-36 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: finishing write tid 3 to smithi20438004-36 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: finishing write tid 4 to smithi20438004-36 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: finishing write tid 5 to smithi20438004-36 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: finishing write tid 6 to smithi20438004-36 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: oid 36 updating version 0 to 57 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: oid 36 updating version 57 to 58 2024-09-13T11:33:02.321 INFO:tasks.rados.rados.0.smithi204.stdout:1197: oid 36 version 58 is already newer than 55 2024-09-13T11:33:02.322 INFO:tasks.rados.rados.0.smithi204.stdout:1197: oid 36 updating version 58 to 59 2024-09-13T11:33:02.322 INFO:tasks.rados.rados.0.smithi204.stdout:1197: oid 36 version 59 is already newer than 56 2024-09-13T11:33:02.322 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 59 (ObjNum 222 snap 0 seq_num 222) dirty exists 2024-09-13T11:33:02.322 INFO:tasks.rados.rados.0.smithi204.stdout:1197: left oid 36 (ObjNum 222 snap 0 seq_num 222) 2024-09-13T11:33:02.322 INFO:tasks.rados.rados.0.smithi204.stdout:1198: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:33:02.328 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:02.328 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:02.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:02.371 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:02.371 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:02.369+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:02.442 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:02.442 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:02.445 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2665684 2024-09-13T11:33:02.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: finishing write tid 1 to smithi20438004-44 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: finishing write tid 2 to smithi20438004-44 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: finishing write tid 3 to smithi20438004-44 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: finishing write tid 4 to smithi20438004-44 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: finishing write tid 5 to smithi20438004-44 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: oid 44 updating version 0 to 46 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: oid 44 version 46 is already newer than 44 2024-09-13T11:33:02.469 INFO:tasks.rados.rados.0.smithi204.stdout:1199: oid 44 version 46 is already newer than 45 2024-09-13T11:33:02.470 INFO:tasks.rados.rados.0.smithi204.stdout:1199: oid 44 version 46 is already newer than 43 2024-09-13T11:33:02.470 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 46 (ObjNum 223 snap 0 seq_num 223) dirty exists 2024-09-13T11:33:02.470 INFO:tasks.rados.rados.0.smithi204.stdout:1199: left oid 44 (ObjNum 223 snap 0 seq_num 223) 2024-09-13T11:33:02.470 INFO:tasks.rados.rados.0.smithi204.stdout:1201: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:02.472 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:02.471+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:02.519 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2858814 2024-09-13T11:33:02.573 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:02.573 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:02.574 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:02.572+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:02.582 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:02.630 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:33:02.653 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2177447 2024-09-13T11:33:02.674 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:02.673+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:02.681 INFO:tasks.rados.rados.0.smithi204.stdout:1200: expect (ObjNum 202 snap 0 seq_num 202) 2024-09-13T11:33:02.687 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:02.687 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:02.696 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:02.732 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:02.775 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:02.774+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:02.798 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:02.800 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:02.800 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:02.809 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1202: finishing set_chunk to oid 26 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 363008 length: 2507776 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 0 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1192: done (10 left) 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1194: done (9 left) 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1195: done (8 left) 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1196: done (7 left) 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1197: done (6 left) 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1198: done (5 left) 2024-09-13T11:33:02.864 INFO:tasks.rados.rados.0.smithi204.stdout:1199: done (4 left) 2024-09-13T11:33:02.865 INFO:tasks.rados.rados.0.smithi204.stdout:1200: done (3 left) 2024-09-13T11:33:02.870 INFO:tasks.rados.rados.0.smithi204.stdout:1201: done (2 left) 2024-09-13T11:33:02.870 INFO:tasks.rados.rados.0.smithi204.stdout:1202: done (1 left) 2024-09-13T11:33:02.870 INFO:tasks.rados.rados.0.smithi204.stdout:1203: done (0 left) 2024-09-13T11:33:02.870 INFO:tasks.rados.rados.0.smithi204.stdout:1204: read oid 11 snap -1 2024-09-13T11:33:02.870 INFO:tasks.rados.rados.0.smithi204.stdout:1204: expect (ObjNum 210 snap 0 seq_num 210) 2024-09-13T11:33:02.870 INFO:tasks.rados.rados.0.smithi204.stdout:1205: set_chunk oid 17 target oid 2024-09-13T11:33:02.876 INFO:tasks.rados.rados.0.smithi204.stdout:1205: set_chunk oid 17 offset: 843264 length: 1270784 target oid d0d9fe376694ee3b92cb2531b265aa9042b0d74cb6627c6387b4b24109fdda51 offset: 0 snap -1 2024-09-13T11:33:02.876 INFO:tasks.rados.rados.0.smithi204.stdout:1206: set_chunk oid 29 target oid 2024-09-13T11:33:02.877 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:02.874+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:02.884 INFO:tasks.rados.rados.0.smithi204.stdout:1206: set_chunk oid 29 offset: 507904 length: 869376 target oid aebd35dd80f1d3e7b56d0c150b78ff9dab61d308788123d7afe4bd3611aa5588 offset: 0 snap -1 2024-09-13T11:33:02.884 INFO:tasks.rados.rados.0.smithi204.stdout:1207: tier_evict oid 28 2024-09-13T11:33:02.885 INFO:tasks.rados.rados.0.smithi204.stdout:1207: tier_evict oid 28 snap -1 2024-09-13T11:33:02.885 INFO:tasks.rados.rados.0.smithi204.stdout:1204: expect (ObjNum 210 snap 0 seq_num 210) 2024-09-13T11:33:02.918 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:02.918 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:02.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:02.976 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:02.974+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:03.035 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:03.036 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:03.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:03.077 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:03.076+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:03.164 INFO:tasks.rados.rados.0.smithi204.stdout:1205: finishing set_chunk to oid 17 2024-09-13T11:33:03.164 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk returned error code -95 offset: 843264 length: 1270784 2024-09-13T11:33:03.164 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-09-13T11:33:03.164 INFO:tasks.rados.rados.0.smithi204.stdout:1207: got (22) Invalid argument 2024-09-13T11:33:03.164 INFO:tasks.rados.rados.0.smithi204.stdout:1206: finishing set_chunk to oid 29 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 89 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1204: done (3 left) 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1205: done (2 left) 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1206: done (1 left) 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1207: done (0 left) 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1208: read oid 29 snap -1 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1208: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1209: read oid 25 snap -1 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1209: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1210: read oid 20 snap -1 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1210: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1211: read oid 35 snap -1 2024-09-13T11:33:03.165 INFO:tasks.rados.rados.0.smithi204.stdout:1211: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:03.166 INFO:tasks.rados.rados.0.smithi204.stdout:1212: read oid 4 snap -1 2024-09-13T11:33:03.166 INFO:tasks.rados.rados.0.smithi204.stdout:1212: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:33:03.166 INFO:tasks.rados.rados.0.smithi204.stdout:1213: set_chunk oid 23 target oid 2024-09-13T11:33:03.178 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:03.177+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:03.192 INFO:tasks.rados.rados.0.smithi204.stdout:1213: set_chunk oid 23 offset: 589312 length: 2573312 target oid 23d788282a3a805fb891b8a9eb64a62047d8580cdaee9e04f45051bb603521af offset: 0 snap -1 2024-09-13T11:33:03.192 INFO:tasks.rados.rados.0.smithi204.stdout:1214: tier_evict oid 33 2024-09-13T11:33:03.192 INFO:tasks.rados.rados.0.smithi204.stdout:1214: tier_evict oid 33 snap -1 2024-09-13T11:33:03.192 INFO:tasks.rados.rados.0.smithi204.stdout:1210: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:03.198 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:03.198 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:03.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:03.226 INFO:teuthology.orchestra.run.smithi038.stdout:get_pg_num_history pg_num_history pg_num_history(e58 pg_nums {1={23=1},2={25=8},3={30=4},4={31=16,56=26}} deleted_pools ) 2024-09-13T11:33:03.227 INFO:teuthology.orchestra.run.smithi038.stdout:Importing pgid 3.3 2024-09-13T11:33:03.227 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:c8f80c84:::bc036ea868a6b2c601148e4e87101fdc0cdd45a73131aa4e7533f4932f45b24d:head# 2024-09-13T11:33:03.227 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.227 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:33:03.228 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.233 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:dd3ad6bf:::4ec2dd001b7153a0bb00c76a8320dec150c3eb87238ccdd9135be5e948aee2d9:head# 2024-09-13T11:33:03.233 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.235 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:de2fec76:::smithi20438004-12 low_tier:head# 2024-09-13T11:33:03.236 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.237 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:e7e3acce:::smithi20438004-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:33:03.238 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.240 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:f1414861:::194f870459df717b3a43948c3b56bc35cac5308afcac3cd76a1a7e1a04723236:head# 2024-09-13T11:33:03.240 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.241 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:33:03.242 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.244 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:f5bbb099:::9b0f46e51af81c905599e26964e04a497e64c0e02a1d69859f9c292a656388d5:head# 2024-09-13T11:33:03.244 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.244 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:33:03.244 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.246 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:33:03.247 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.249 INFO:teuthology.orchestra.run.smithi038.stdout:Write #3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head# 2024-09-13T11:33:03.249 INFO:teuthology.orchestra.run.smithi038.stdout:snapset 0=[]:{} 2024-09-13T11:33:03.251 INFO:teuthology.orchestra.run.smithi038.stdout:write_pg epoch 39 info 3.3( v 39'53 (39'50,39'53] local-lis/les=30/31 n=11 ec=30/30 lis/c=30/30 les/c/f=31/31/0 sis=30) 2024-09-13T11:33:03.251 INFO:teuthology.orchestra.run.smithi038.stdout:Import successful 2024-09-13T11:33:03.280 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:03.278+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:03.306 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2829032 2024-09-13T11:33:03.313 INFO:tasks.rados.rados.0.smithi204.stdout:1212: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:33:03.317 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:03.318 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:03.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:03.346 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:03.368 INFO:tasks.rados.rados.0.smithi204.stdout:1211: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:03.368 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:03.381 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:03.380+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:03.469 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:03.476 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:03.477 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:03.483 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:03.481+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:03.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:03.528 INFO:tasks.rados.rados.0.smithi204.stdout:1208: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:03.542 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1283535 2024-09-13T11:33:03.584 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:03.582+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:03.592 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:03.593 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:03.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:03.685 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:03.684+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:03.704 INFO:tasks.rados.rados.0.smithi204.stdout:1209: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:33:03.704 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:03.749 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:03.749 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:03.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:03.768 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:03.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /var/log/ceph/exp.3.3.2 2024-09-13T11:33:03.786 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:03.785+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:03.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /var/log/ceph/exp.3.3.2 2024-09-13T11:33:03.866 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:03.866 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:03.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:03.887 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:03.886+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1214: got (22) Invalid argument 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1213: finishing set_chunk to oid 23 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 75 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1208: done (6 left) 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1209: done (5 left) 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1210: done (4 left) 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1211: done (3 left) 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1212: done (2 left) 2024-09-13T11:33:03.900 INFO:tasks.rados.rados.0.smithi204.stdout:1213: done (1 left) 2024-09-13T11:33:03.901 INFO:tasks.rados.rados.0.smithi204.stdout:1214: done (0 left) 2024-09-13T11:33:03.901 INFO:tasks.rados.rados.0.smithi204.stdout:1215: read oid 38 snap -1 2024-09-13T11:33:03.901 INFO:tasks.rados.rados.0.smithi204.stdout:1215: expect (ObjNum 214 snap 0 seq_num 214) 2024-09-13T11:33:03.901 INFO:tasks.rados.rados.0.smithi204.stdout:1216: read oid 30 snap -1 2024-09-13T11:33:03.901 INFO:tasks.rados.rados.0.smithi204.stdout:1216: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:33:03.901 INFO:tasks.rados.rados.0.smithi204.stdout:1217: write (excl) oid 12 current snap is 0 2024-09-13T11:33:03.901 INFO:tasks.rados.rados.0.smithi204.stdout:1217: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-13T11:33:03.907 INFO:tasks.rados.rados.0.smithi204.stdout:1217: writing smithi20438004-12 from 581575 to 1286837 tid 1 2024-09-13T11:33:03.913 INFO:tasks.rados.rados.0.smithi204.stdout:1217: writing smithi20438004-12 from 1733272 to 2440681 tid 2 2024-09-13T11:33:03.921 INFO:tasks.rados.rados.0.smithi204.stdout:1217: writing smithi20438004-12 from 3195938 to 3633435 tid 3 2024-09-13T11:33:03.921 INFO:tasks.rados.rados.0.smithi204.stdout:1218: read oid 47 snap -1 2024-09-13T11:33:03.921 INFO:tasks.rados.rados.0.smithi204.stdout:1218: expect deleted 2024-09-13T11:33:03.921 INFO:tasks.rados.rados.0.smithi204.stdout:1219: write oid 21 current snap is 0 2024-09-13T11:33:03.921 INFO:tasks.rados.rados.0.smithi204.stdout:1219: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-13T11:33:03.928 INFO:tasks.rados.rados.0.smithi204.stdout:1219: writing smithi20438004-21 from 598382 to 1378893 tid 1 2024-09-13T11:33:03.934 INFO:tasks.rados.rados.0.smithi204.stdout:1219: writing smithi20438004-21 from 2075401 to 2526468 tid 2 2024-09-13T11:33:03.941 INFO:tasks.rados.rados.0.smithi204.stdout:1219: writing smithi20438004-21 from 3107008 to 3755777 tid 3 2024-09-13T11:33:03.941 INFO:tasks.rados.rados.0.smithi204.stdout:1220: tier_flush oid 39 2024-09-13T11:33:03.941 INFO:tasks.rados.rados.0.smithi204.stdout:1220: tier_flush oid 39 snap -1 2024-09-13T11:33:03.941 INFO:tasks.rados.rados.0.smithi204.stdout:1215: expect (ObjNum 214 snap 0 seq_num 214) 2024-09-13T11:33:03.988 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:03.987+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:04.023 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:04.023 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:04.032 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:04.088 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:04.087+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:04.146 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:04.146 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:04.155 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:04.190 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:04.189+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:04.213 INFO:tasks.rados.rados.0.smithi204.stdout:1216: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-13T11:33:04.223 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1389520 2024-09-13T11:33:04.236 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:04.259 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:04.259 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:04.267 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:04.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:04.285 INFO:tasks.rados.rados.0.smithi204.stdout:1217: finishing write tid 1 to smithi20438004-12 2024-09-13T11:33:04.286 INFO:tasks.rados.rados.0.smithi204.stdout:1217: finishing write tid 2 to smithi20438004-12 2024-09-13T11:33:04.286 INFO:tasks.rados.rados.0.smithi204.stdout:1215: done (5 left) 2024-09-13T11:33:04.286 INFO:tasks.rados.rados.0.smithi204.stdout:1216: done (4 left) 2024-09-13T11:33:04.286 INFO:tasks.rados.rados.0.smithi204.stdout:1218: done (3 left) 2024-09-13T11:33:04.286 INFO:tasks.rados.rados.0.smithi204.stdout:1221: write (excl) oid 38 current snap is 0 2024-09-13T11:33:04.286 INFO:tasks.rados.rados.0.smithi204.stdout:1221: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-13T11:33:04.291 INFO:tasks.rados.rados.0.smithi204.stdout:1221: writing smithi20438004-38 from 615189 to 1070949 tid 1 2024-09-13T11:33:04.302 INFO:tasks.rados.rados.0.smithi204.stdout:1221: writing smithi20438004-38 from 1733883 to 2444961 tid 2 2024-09-13T11:33:04.306 INFO:tasks.rados.rados.0.smithi204.stdout:1221: writing smithi20438004-38 from 3134431 to 3594472 tid 3 2024-09-13T11:33:04.306 INFO:tasks.rados.rados.0.smithi204.stdout:1222: write (excl) oid 43 current snap is 0 2024-09-13T11:33:04.306 INFO:tasks.rados.rados.0.smithi204.stdout:1217: finishing write tid 3 to smithi20438004-12 2024-09-13T11:33:04.306 INFO:tasks.rados.rados.0.smithi204.stdout:1217: finishing write tid 4 to smithi20438004-12 2024-09-13T11:33:04.306 INFO:tasks.rados.rados.0.smithi204.stdout:1217: finishing write tid 5 to smithi20438004-12 2024-09-13T11:33:04.306 INFO:tasks.rados.rados.0.smithi204.stdout:1217: oid 12 updating version 0 to 69 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:1217: oid 12 version 69 is already newer than 68 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:1217: oid 12 version 69 is already newer than 66 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:1217: oid 12 version 69 is already newer than 67 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 69 (ObjNum 225 snap 0 seq_num 225) dirty exists 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:1217: left oid 12 (ObjNum 225 snap 0 seq_num 225) 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:1219: finishing write tid 1 to smithi20438004-21 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:1219: finishing write tid 2 to smithi20438004-21 2024-09-13T11:33:04.307 INFO:tasks.rados.rados.0.smithi204.stdout:1222: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-13T11:33:04.311 INFO:tasks.rados.rados.0.smithi204.stdout:1222: writing smithi20438004-43 from 631996 to 1163005 tid 1 2024-09-13T11:33:04.317 INFO:tasks.rados.rados.0.smithi204.stdout:1222: writing smithi20438004-43 from 1792365 to 2247101 tid 2 2024-09-13T11:33:04.324 INFO:tasks.rados.rados.0.smithi204.stdout:1222: writing smithi20438004-43 from 2761854 to 3433167 tid 3 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1217: done (4 left) 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1223: write oid 11 current snap is 0 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: finishing write tid 3 to smithi20438004-21 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: finishing write tid 4 to smithi20438004-21 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: finishing write tid 5 to smithi20438004-21 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: oid 21 updating version 0 to 29 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: oid 21 updating version 29 to 31 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: oid 21 version 31 is already newer than 30 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: oid 21 version 31 is already newer than 28 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 31 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1219: left oid 21 (ObjNum 226 snap 0 seq_num 226) 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:1220: got (0) Success 2024-09-13T11:33:04.325 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-13T11:33:04.326 INFO:tasks.rados.rados.0.smithi204.stdout:1223: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-13T11:33:04.328 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:04.326+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:04.332 INFO:tasks.rados.rados.0.smithi204.stdout:1223: writing smithi20438004-11 from 648803 to 1371414 tid 1 2024-09-13T11:33:04.339 INFO:tasks.rados.rados.0.smithi204.stdout:1223: writing smithi20438004-11 from 1967200 to 2681947 tid 2 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1223: writing smithi20438004-11 from 3421983 to 3848803 tid 3 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1219: done (4 left) 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1220: done (3 left) 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1224: read oid 5 snap -1 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1224: expect (ObjNum 181 snap 0 seq_num 181) 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1225: read oid 25 snap -1 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1225: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:33:04.346 INFO:tasks.rados.rados.0.smithi204.stdout:1226: set_chunk oid 30 target oid 2024-09-13T11:33:04.358 INFO:tasks.rados.rados.0.smithi204.stdout:1226: set_chunk oid 30 offset: 98304 length: 1033728 target oid 2cf6adb8b48e081b3febe5b5f21de25e25050ea7f7e674a277fe509d132aa231 offset: 0 snap -1 2024-09-13T11:33:04.358 INFO:tasks.rados.rados.0.smithi204.stdout:1227: read oid 46 snap -1 2024-09-13T11:33:04.358 INFO:tasks.rados.rados.0.smithi204.stdout:1227: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:04.358 INFO:tasks.rados.rados.0.smithi204.stdout:1221: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1228: read oid 32 snap -1 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1228: expect (ObjNum 202 snap 0 seq_num 202) 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1229: read oid 41 snap -1 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1229: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1230: read oid 40 snap -1 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1230: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1231: read oid 49 snap -1 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1231: expect deleted 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1232: tier_flush oid 12 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1232: tier_flush oid 12 snap -1 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1233: read oid 48 snap -1 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1233: expect (ObjNum 188 snap 0 seq_num 188) 2024-09-13T11:33:04.359 INFO:tasks.rados.rados.0.smithi204.stdout:1234: set_chunk oid 24 target oid 2024-09-13T11:33:04.368 INFO:tasks.rados.rados.0.smithi204.stdout:1234: set_chunk oid 24 offset: 2338304 length: 157184 target oid 83ead958e52787e8c61ccf7552389eb7dc83aad705d08efffa6f16d1dc689235 offset: 0 snap -1 2024-09-13T11:33:04.368 INFO:tasks.rados.rados.0.smithi204.stdout:1235: read oid 15 snap -1 2024-09-13T11:33:04.368 INFO:tasks.rados.rados.0.smithi204.stdout:1235: expect deleted 2024-09-13T11:33:04.368 INFO:tasks.rados.rados.0.smithi204.stdout:1236: read oid 29 snap -1 2024-09-13T11:33:04.368 INFO:tasks.rados.rados.0.smithi204.stdout:1236: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: oid 38 updating version 0 to 101 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: oid 38 version 101 is already newer than 100 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: oid 38 version 101 is already newer than 99 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: oid 38 updating version 101 to 102 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 102 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: left oid 38 (ObjNum 227 snap 0 seq_num 227) 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1222: finishing write tid 1 to smithi20438004-43 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1221: done (15 left) 2024-09-13T11:33:04.369 INFO:tasks.rados.rados.0.smithi204.stdout:1237: read oid 23 snap -1 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1237: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: finishing write tid 2 to smithi20438004-43 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: finishing write tid 3 to smithi20438004-43 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: finishing write tid 4 to smithi20438004-43 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: finishing write tid 5 to smithi20438004-43 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: oid 43 updating version 0 to 109 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: oid 43 version 109 is already newer than 108 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: oid 43 version 109 is already newer than 107 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: oid 43 updating version 109 to 110 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 110 (ObjNum 228 snap 0 seq_num 228) dirty exists 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: left oid 43 (ObjNum 228 snap 0 seq_num 228) 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1223: finishing write tid 1 to smithi20438004-11 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1222: done (15 left) 2024-09-13T11:33:04.370 INFO:tasks.rados.rados.0.smithi204.stdout:1238: read oid 4 snap -1 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1238: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: finishing write tid 2 to smithi20438004-11 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: finishing write tid 3 to smithi20438004-11 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: finishing write tid 4 to smithi20438004-11 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: finishing write tid 5 to smithi20438004-11 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: oid 11 updating version 0 to 104 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: oid 11 updating version 104 to 105 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: oid 11 version 105 is already newer than 103 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: oid 11 updating version 105 to 106 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 106 (ObjNum 229 snap 0 seq_num 229) dirty exists 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: left oid 11 (ObjNum 229 snap 0 seq_num 229) 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1223: done (15 left) 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1231: done (14 left) 2024-09-13T11:33:04.371 INFO:tasks.rados.rados.0.smithi204.stdout:1239: tier_promote oid 8 2024-09-13T11:33:04.372 INFO:tasks.rados.rados.0.smithi204.stdout:1230: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:04.372 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:04.372 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:04.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:04.392 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:04.390+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:04.414 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:04.415 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:04.432 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:04.432 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:04.449 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:04.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:04.465 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:04.466 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:04.481 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2291883 2024-09-13T11:33:04.482 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:04.482 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:04.484 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:04.484 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:04.484 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:04.484 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:04.492 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:04.491+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:04.493 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:04.494 INFO:tasks.rados.rados.0.smithi204.stdout:1224: expect (ObjNum 181 snap 0 seq_num 181) 2024-09-13T11:33:04.494 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 272010 2024-09-13T11:33:04.499 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:04.500 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:04.516 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:04.516 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:04.533 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:04.533 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:04.549 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:04.550 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:04.567 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:04.584 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:04.584 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:04.584 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:04.594 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:04.592+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:04.596 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:04.596 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:04.596 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:04.596 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:04.596 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:04.601 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:04.601 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:04.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:04.618 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:04.618 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:04.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:04.636 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:04.640 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:04.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 58.364792139999999, 2024-09-13T11:33:04.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079674329999999995, 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4344", 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343169+0000", 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343177+0000", 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4819999999999994e-06 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343189+0000", 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2306999999999999e-05 2024-09-13T11:33:04.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343199+0000", 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343279+0000", 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0025999999999996e-05 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343289+0000", 2024-09-13T11:33:04.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0518e-05 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348428+0000", 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051380779999999999 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348510+0000", 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2226999999999997e-05 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348598+0000", 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8722999999999997e-05 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349014+0000", 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041537900000000001 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349279+0000", 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026517700000000001 2024-09-13T11:33:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351081+0000", 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001802292 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351110+0000", 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8651000000000001e-05 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351132+0000", 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069000000000002e-05 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:21 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'4, mlcod=38'2)", 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.194809088, 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002707297, 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513149+0000", 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.322e-06 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514452+0000", 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303153 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514468+0000", 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6045999999999998e-05 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514478+0000", 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4960000000000002e-06 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514531+0000", 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3442000000000001e-05 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514546+0000", 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5211999999999999e-05 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515821+0000", 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001274854 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515855+0000", 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3772000000000001e-05 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.68658259, 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002340637, 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021376+0000", 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.013e-06 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022454+0000", 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010781110000000001 2024-09-13T11:33:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022469+0000", 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4199000000000001e-05 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022477+0000", 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022532+0000", 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5636000000000001e-05 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022548+0000", 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5769000000000001e-05 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023686+0000", 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011379929999999999 2024-09-13T11:33:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023715+0000", 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8659000000000001e-05 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:182 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'8, mlcod=38'6)", 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.680516916999999, 2024-09-13T11:33:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022898689999999999, 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027441+0000", 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7599999999999996e-07 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028556+0000", 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011154439999999999 2024-09-13T11:33:04.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028566+0000", 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8269999999999993e-06 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028571+0000", 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8550000000000001e-06 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028614+0000", 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028629+0000", 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029703+0000", 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001073898 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029730+0000", 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6993999999999999e-05 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:189 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'10, mlcod=38'8)", 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.658095613999997, 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020004459999999999, 2024-09-13T11:33:04.732 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049862+0000", 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2999999999999999e-07 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050772+0000", 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091040000000000001 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050787+0000", 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4021e-05 2024-09-13T11:33:04.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050793+0000", 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6959999999999999e-06 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050843+0000", 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0148999999999997e-05 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050860+0000", 2024-09-13T11:33:04.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6665999999999999e-05 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.051831+0000", 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097091600000000005 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.051862+0000", 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0667999999999998e-05 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.736 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:190 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'11, mlcod=38'9)", 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.650689188000001, 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022963139999999998, 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058460+0000", 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192238 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058470+0000", 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058475+0000", 2024-09-13T11:33:04.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5399999999999997e-06 2024-09-13T11:33:04.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058514+0000", 2024-09-13T11:33:04.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8887999999999998e-05 2024-09-13T11:33:04.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058526+0000", 2024-09-13T11:33:04.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1683e-05 2024-09-13T11:33:04.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059537+0000", 2024-09-13T11:33:04.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001011907 2024-09-13T11:33:04.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059564+0000", 2024-09-13T11:33:04.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:33:04.743 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.743 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.313505880999998, 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048595679999999999, 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394452+0000", 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-13T11:33:04.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397390+0000", 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002937977 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397401+0000", 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1172000000000001e-05 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397407+0000", 2024-09-13T11:33:04.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:04.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397452+0000", 2024-09-13T11:33:04.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4705e-05 2024-09-13T11:33:04.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397467+0000", 2024-09-13T11:33:04.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5622000000000001e-05 2024-09-13T11:33:04.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399273+0000", 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001805823 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399310+0000", 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7512999999999998e-05 2024-09-13T11:33:04.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.750 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.750 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.750 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1363 4.2 e43/31 4:4ba9f189:::smithi20438004-14:head v 43'50, mlcod=43'48)", 2024-09-13T11:33:04.750 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:04.751 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.662373448, 2024-09-13T11:33:04.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021379849999999998, 2024-09-13T11:33:04.751 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.751 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.751 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.751 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:04.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:04.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045585+0000", 2024-09-13T11:33:04.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.604e-06 2024-09-13T11:33:04.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045958+0000", 2024-09-13T11:33:04.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000372672 2024-09-13T11:33:04.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045978+0000", 2024-09-13T11:33:04.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0262999999999999e-05 2024-09-13T11:33:04.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045991+0000", 2024-09-13T11:33:04.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2811e-05 2024-09-13T11:33:04.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046079+0000", 2024-09-13T11:33:04.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8667999999999999e-05 2024-09-13T11:33:04.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046103+0000", 2024-09-13T11:33:04.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4131e-05 2024-09-13T11:33:04.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047664+0000", 2024-09-13T11:33:04.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001560098 2024-09-13T11:33:04.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047721+0000", 2024-09-13T11:33:04.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7738000000000002e-05 2024-09-13T11:33:04.760 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.760 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.761 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.761 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1410 4.4 e43/41 4:2451796c:::smithi20438004-17:head v 43'44, mlcod=43'42)", 2024-09-13T11:33:04.761 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:04.762 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.579400939999999, 2024-09-13T11:33:04.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022888489999999999, 2024-09-13T11:33:04.762 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.762 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.762 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.762 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:04.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:04.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128557+0000", 2024-09-13T11:33:04.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3939999999999999e-06 2024-09-13T11:33:04.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129014+0000", 2024-09-13T11:33:04.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045637100000000001 2024-09-13T11:33:04.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129032+0000", 2024-09-13T11:33:04.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8685000000000002e-05 2024-09-13T11:33:04.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129042+0000", 2024-09-13T11:33:04.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-09-13T11:33:04.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129100+0000", 2024-09-13T11:33:04.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8238e-05 2024-09-13T11:33:04.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129119+0000", 2024-09-13T11:33:04.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8258e-05 2024-09-13T11:33:04.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130809+0000", 2024-09-13T11:33:04.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001690045 2024-09-13T11:33:04.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130845+0000", 2024-09-13T11:33:04.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5991000000000002e-05 2024-09-13T11:33:04.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.772 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.773 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.773 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:04.773 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:04.774 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.676811034, 2024-09-13T11:33:04.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071247150000000002, 2024-09-13T11:33:04.774 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.774 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.774 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.774 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:04.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:04.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031147+0000", 2024-09-13T11:33:04.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.579e-06 2024-09-13T11:33:04.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036431+0000", 2024-09-13T11:33:04.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00528394 2024-09-13T11:33:04.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036449+0000", 2024-09-13T11:33:04.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7329e-05 2024-09-13T11:33:04.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036457+0000", 2024-09-13T11:33:04.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:04.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036529+0000", 2024-09-13T11:33:04.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1506000000000003e-05 2024-09-13T11:33:04.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036551+0000", 2024-09-13T11:33:04.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2154000000000002e-05 2024-09-13T11:33:04.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038234+0000", 2024-09-13T11:33:04.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001683192 2024-09-13T11:33:04.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038270+0000", 2024-09-13T11:33:04.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6408000000000001e-05 2024-09-13T11:33:04.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.783 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.784 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1468 4.4 e44/41 4:2b094039:::smithi20438004-40:head v 44'48, mlcod=43'46)", 2024-09-13T11:33:04.784 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:33:04.784 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.638962119999999, 2024-09-13T11:33:04.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020495919999999998, 2024-09-13T11:33:04.785 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.785 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.785 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.785 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:33:04.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:33:04.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068996+0000", 2024-09-13T11:33:04.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4649999999999999e-06 2024-09-13T11:33:04.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069273+0000", 2024-09-13T11:33:04.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027673599999999997 2024-09-13T11:33:04.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069292+0000", 2024-09-13T11:33:04.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9202999999999999e-05 2024-09-13T11:33:04.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069302+0000", 2024-09-13T11:33:04.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:33:04.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069367+0000", 2024-09-13T11:33:04.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5112999999999998e-05 2024-09-13T11:33:04.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069393+0000", 2024-09-13T11:33:04.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5867000000000002e-05 2024-09-13T11:33:04.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.070994+0000", 2024-09-13T11:33:04.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016011980000000001 2024-09-13T11:33:04.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.071044+0000", 2024-09-13T11:33:04.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0421999999999999e-05 2024-09-13T11:33:04.794 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.794 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.794 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.795 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1679 4.4 4.9c0290d4 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:33:04.795 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:04.795 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.648753330000002, 2024-09-13T11:33:04.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47635528100000002, 2024-09-13T11:33:04.796 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.796 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.796 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:04.796 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:04.796 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:04.796 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:04.797 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1679 2024-09-13T11:33:04.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.797 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:04.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:04.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059205+0000", 2024-09-13T11:33:04.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.812e-06 2024-09-13T11:33:04.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059211+0000", 2024-09-13T11:33:04.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1469999999999998e-06 2024-09-13T11:33:04.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059222+0000", 2024-09-13T11:33:04.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0560000000000001e-05 2024-09-13T11:33:04.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059233+0000", 2024-09-13T11:33:04.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0591000000000001e-05 2024-09-13T11:33:04.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059286+0000", 2024-09-13T11:33:04.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519e-05 2024-09-13T11:33:04.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:04.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059293+0000", 2024-09-13T11:33:04.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4690000000000002e-06 2024-09-13T11:33:04.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535521+0000", 2024-09-13T11:33:04.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47622830100000002 2024-09-13T11:33:04.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535559+0000", 2024-09-13T11:33:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7882000000000001e-05 2024-09-13T11:33:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:33:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.305505700000001, 2024-09-13T11:33:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002436952, 2024-09-13T11:33:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402453+0000", 2024-09-13T11:33:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.632e-06 2024-09-13T11:33:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403268+0000", 2024-09-13T11:33:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081553400000000003 2024-09-13T11:33:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403289+0000", 2024-09-13T11:33:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0353e-05 2024-09-13T11:33:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403298+0000", 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4760000000000004e-06 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403369+0000", 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0858000000000005e-05 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403397+0000", 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8518e-05 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404841+0000", 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001443228 2024-09-13T11:33:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404888+0000", 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7352999999999999e-05 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.299979304000001, 2024-09-13T11:33:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003287178, 2024-09-13T11:33:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407979+0000", 2024-09-13T11:33:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-09-13T11:33:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409585+0000", 2024-09-13T11:33:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016060479999999999 2024-09-13T11:33:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409598+0000", 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3027000000000001e-05 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409606+0000", 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2190000000000004e-06 2024-09-13T11:33:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409676+0000", 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9791999999999999e-05 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409707+0000", 2024-09-13T11:33:04.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1692000000000001e-05 2024-09-13T11:33:04.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411223+0000", 2024-09-13T11:33:04.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001515705 2024-09-13T11:33:04.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411265+0000", 2024-09-13T11:33:04.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1649000000000001e-05 2024-09-13T11:33:04.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.825 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.825 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1705 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'62, mlcod=0'0)", 2024-09-13T11:33:04.826 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:04.826 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.292374903999999, 2024-09-13T11:33:04.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003138161, 2024-09-13T11:33:04.826 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.826 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.826 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.826 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:04.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:04.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415583+0000", 2024-09-13T11:33:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9699999999999995e-07 2024-09-13T11:33:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416900+0000", 2024-09-13T11:33:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001317251 2024-09-13T11:33:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416926+0000", 2024-09-13T11:33:04.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5622e-05 2024-09-13T11:33:04.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416942+0000", 2024-09-13T11:33:04.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6741000000000001e-05 2024-09-13T11:33:04.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417014+0000", 2024-09-13T11:33:04.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1298999999999996e-05 2024-09-13T11:33:04.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417042+0000", 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.851e-05 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418673+0000", 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016310949999999999 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418720+0000", 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6845999999999998e-05 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1706 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'63, mlcod=45'61)", 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.287042707000001, 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003090597, 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.834 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420915+0000", 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-06 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422306+0000", 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001390753 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422317+0000", 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422322+0000", 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4360000000000001e-06 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422379+0000", 2024-09-13T11:33:04.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7003000000000003e-05 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422405+0000", 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6424e-05 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.423958+0000", 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015526680000000001 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.424005+0000", 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6619000000000002e-05 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1919 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'63, mlcod=53'61)", 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.741842729, 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002758908, 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.839 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966116+0000", 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0370000000000001e-06 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967135+0000", 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010185260000000001 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967149+0000", 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3903999999999999e-05 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967157+0000", 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.405e-06 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967255+0000", 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8441000000000002e-05 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967280+0000", 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5079000000000002e-05 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968828+0000", 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001547936 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968873+0000", 2024-09-13T11:33:04.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4579999999999997e-05 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.733892006, 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028733999999999999, 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.844 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974066+0000", 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975332+0000", 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012659349999999999 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975345+0000", 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3402e-05 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975351+0000", 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4450000000000001e-06 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975438+0000", 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6687999999999994e-05 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975457+0000", 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8913e-05 2024-09-13T11:33:04.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976901+0000", 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014437180000000001 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976938+0000", 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7441999999999998e-05 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.788758241, 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022722770000000001, 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919200+0000", 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9000000000000005e-07 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919903+0000", 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070307200000000001 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919915+0000", 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2482e-05 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919922+0000", 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5459999999999997e-06 2024-09-13T11:33:04.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919988+0000", 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6377999999999994e-05 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.920027+0000", 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8649999999999998e-05 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921419+0000", 2024-09-13T11:33:04.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013924530000000001 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921471+0000", 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1706000000000003e-05 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2077 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'67, mlcod=53'65)", 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.78061561, 2024-09-13T11:33:04.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026288430000000001, 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:04.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927342+0000", 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928463+0000", 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011212640000000001 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928476+0000", 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3147e-05 2024-09-13T11:33:04.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928484+0000", 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928549+0000", 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5001999999999999e-05 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:04.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928565+0000", 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6124999999999999e-05 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929914+0000", 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001348859 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929970+0000", 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5912000000000003e-05 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.857 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:04.858 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:04.858 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:04.859 INFO:tasks.rados.rados.0.smithi204.stdout:1227: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:04.859 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:04.859 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:04.722+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:04.860 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:04.857+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:04.860 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:04.918 INFO:tasks.rados.rados.0.smithi204.stdout:1229: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-13T11:33:04.960 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:04.958+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:04.963 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:04.963 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:04.964 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:04.964 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:04.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:05.045 INFO:tasks.rados.rados.0.smithi204.stdout:1233: expect (ObjNum 188 snap 0 seq_num 188) 2024-09-13T11:33:05.061 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:05.059+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:05.067 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:05.075 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:05.075 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:05.075 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:05.075 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:05.075 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:05.083 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:05.109 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1253508 2024-09-13T11:33:05.131 INFO:tasks.rados.rados.0.smithi204.stdout:1239: got (0) Success 2024-09-13T11:33:05.131 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 28 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-13T11:33:05.131 INFO:tasks.rados.rados.0.smithi204.stdout:1228: expect (ObjNum 202 snap 0 seq_num 202) 2024-09-13T11:33:05.131 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:05.168 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:05.167+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:05.194 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 303839 2024-09-13T11:33:05.228 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:05.228 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:05.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:05.228 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:206 3.1 e38/30 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.134092461999998, 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012415890000000001, 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092192+0000", 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.63e-07 2024-09-13T11:33:05.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100358+0000", 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081664059999999993 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100384+0000", 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5287e-05 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100401+0000", 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7744000000000002e-05 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100478+0000", 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6769999999999999e-05 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100508+0000", 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0134e-05 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104514+0000", 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040060590000000002 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104607+0000", 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3027000000000002e-05 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:201 3.1 e38/30 3:b265d015:::smithi20438004-2 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:05.234 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.125847123, 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012387119, 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100438+0000", 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1000000000000001e-06 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108964+0000", 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085255959999999999 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108984+0000", 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0466e-05 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108996+0000", 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2149e-05 2024-09-13T11:33:05.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109097+0000", 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010042799999999999 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109121+0000", 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.455e-05 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112773+0000", 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036519780000000002 2024-09-13T11:33:05.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112824+0000", 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0852e-05 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:05.239 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.117225652000002, 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096300529999999995, 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109060+0000", 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116086+0000", 2024-09-13T11:33:05.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070260239999999996 2024-09-13T11:33:05.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116100+0000", 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4209999999999999e-05 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116109+0000", 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6569999999999992e-06 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116160+0000", 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0470000000000003e-05 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116191+0000", 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1494e-05 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118639+0000", 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024481030000000001 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118688+0000", 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8866999999999998e-05 2024-09-13T11:33:05.244 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.110153857, 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010922463, 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:05.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:05.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116131+0000", 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6899999999999998e-07 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124407+0000", 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082764919999999999 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124422+0000", 2024-09-13T11:33:05.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5404999999999999e-05 2024-09-13T11:33:05.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124432+0000", 2024-09-13T11:33:05.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8910000000000007e-06 2024-09-13T11:33:05.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124485+0000", 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2750999999999997e-05 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124508+0000", 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3076999999999999e-05 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127004+0000", 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024962930000000001 2024-09-13T11:33:05.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127053+0000", 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8084999999999998e-05 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:05.253 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.096489759000001, 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091903760000000001, 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129795+0000", 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8700000000000001e-07 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135325+0000", 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055302190000000003 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135340+0000", 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135351+0000", 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1358999999999999e-05 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135418+0000", 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000002e-05 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135447+0000", 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8415000000000002e-05 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138936+0000", 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034896800000000002 2024-09-13T11:33:05.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138985+0000", 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8338999999999999e-05 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:05.258 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.086294686999999, 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010981263, 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139990+0000", 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2e-07 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147357+0000", 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073669950000000003 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147376+0000", 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9453000000000002e-05 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147388+0000", 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1527e-05 2024-09-13T11:33:05.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147472+0000", 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4630999999999997e-05 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147493+0000", 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0687e-05 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150899+0000", 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034054739999999999 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150971+0000", 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1976000000000003e-05 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.078843679000002, 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015742025999999999, 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147441+0000", 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7700000000000004e-07 2024-09-13T11:33:05.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157019+0000", 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095782090000000007 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157057+0000", 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157072+0000", 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5082999999999999e-05 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157157+0000", 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5065e-05 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157188+0000", 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0920000000000002e-05 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163133+0000", 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059446890000000004 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163182+0000", 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9564999999999997e-05 2024-09-13T11:33:05.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.068771591000001, 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015556076, 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.269 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157514+0000", 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167722+0000", 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010208016 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167740+0000", 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7668000000000001e-05 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167754+0000", 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4647999999999999e-05 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167831+0000", 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7229000000000002e-05 2024-09-13T11:33:05.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167855+0000", 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3467000000000001e-05 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173001+0000", 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051458160000000001 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173068+0000", 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7720999999999998e-05 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.058503098999999, 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011370243, 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:05.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3499999999999996e-07 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175562+0000", 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077811060000000003 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175578+0000", 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175587+0000", 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175668+0000", 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1224999999999996e-05 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175695+0000", 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6505000000000001e-05 2024-09-13T11:33:05.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179075+0000", 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003380508 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179151+0000", 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5927000000000007e-05 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.050676438000004, 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017514140000000001, 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188284+0000", 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012675702 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188299+0000", 2024-09-13T11:33:05.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5122e-05 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188308+0000", 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9819999999999997e-06 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188367+0000", 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8774999999999999e-05 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188387+0000", 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0649000000000002e-05 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193056+0000", 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046684830000000002 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193122+0000", 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5790999999999994e-05 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.037951460000002, 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015855429000000001, 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.285 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188333+0000", 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-07 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199600+0000", 2024-09-13T11:33:05.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011266717000000001 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199613+0000", 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3682e-05 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199620+0000", 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1339999999999998e-06 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199676+0000", 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5689000000000002e-05 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199708+0000", 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2082999999999998e-05 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204123+0000", 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004414734 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204188+0000", 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.026642922999997, 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092010560000000009, 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199642+0000", 2024-09-13T11:33:05.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9599999999999999e-07 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205539+0000", 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058972290000000004 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205553+0000", 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4577e-05 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205564+0000", 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0254e-05 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205630+0000", 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6057999999999997e-05 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205658+0000", 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7979000000000001e-05 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208792+0000", 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031342499999999999 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208842+0000", 2024-09-13T11:33:05.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0213000000000002e-05 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:245 3.1 e38/30 3:adf25eb4:::smithi20438004-46 low_tier:head v 38'15, mlcod=38'11)", 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.020696594, 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011146911000000001, 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.296 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205588+0000", 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213344+0000", 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077554440000000002 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213359+0000", 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5455999999999999e-05 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213369+0000", 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9890000000000004e-06 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213453+0000", 2024-09-13T11:33:05.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4153999999999994e-05 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213479+0000", 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5831999999999999e-05 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216694+0000", 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032149990000000001 2024-09-13T11:33:05.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216734+0000", 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0102999999999999e-05 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:260 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.301 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.006925572999997, 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011239344, 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 260 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.302 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219360+0000", 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219366+0000", 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7179999999999999e-06 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219376+0000", 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9990000000000003e-06 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219382+0000", 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8150000000000002e-06 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219428+0000", 2024-09-13T11:33:05.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5751999999999998e-05 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4119000000000001e-05 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227887+0000", 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083953989999999996 2024-09-13T11:33:05.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228998+0000", 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001110764 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229290+0000", 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029145400000000002 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230489+0000", 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001199786 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230548+0000", 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8900999999999997e-05 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230598+0000", 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9391999999999997e-05 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:249 3.1 e38/30 3:81faf980:::smithi20438004-50 low_tier:head v 38'21, mlcod=38'18)", 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.003649437, 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091134460000000007, 2024-09-13T11:33:05.309 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:05.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.68e-07 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227843+0000", 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005207494 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227853+0000", 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117e-05 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227860+0000", 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4880000000000001e-06 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227912+0000", 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1900000000000001e-05 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227933+0000", 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0474000000000001e-05 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231709+0000", 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037762759999999999 2024-09-13T11:33:05.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231748+0000", 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9029000000000002e-05 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:278 4.2 4:4d4c8512:::smithi20438004-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.314 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.995289837000001, 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091148359999999994, 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230995+0000", 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4200000000000004e-07 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231000+0000", 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3290000000000001e-06 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231017+0000", 2024-09-13T11:33:05.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6242000000000001e-05 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231025+0000", 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231094+0000", 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9554999999999994e-05 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231171+0000", 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7082999999999994e-05 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236799+0000", 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056282260000000001 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237539+0000", 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073930700000000003 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237574+0000", 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5191999999999998e-05 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240013+0000", 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002439185 2024-09-13T11:33:05.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240070+0000", 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6650999999999997e-05 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240109+0000", 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9131999999999997e-05 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.321 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:287 4.2 4:4297cb20:::smithi20438004-38:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.991354364999999, 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013317569, 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.322 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 287 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234931+0000", 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.192e-06 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234936+0000", 2024-09-13T11:33:05.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0640000000000001e-06 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234953+0000", 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7292e-05 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234961+0000", 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:33:05.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235026+0000", 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5073000000000005e-05 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235111+0000", 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4635999999999994e-05 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242149+0000", 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070384990000000001 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069074700000000002 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242936+0000", 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5730000000000007e-05 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248138+0000", 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052018330000000003 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248197+0000", 2024-09-13T11:33:05.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9512000000000002e-05 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248247+0000", 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.329 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:310 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.977427996999999, 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095515259999999994, 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 310 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.330 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248857+0000", 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-06 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248863+0000", 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6529999999999999e-06 2024-09-13T11:33:05.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248874+0000", 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.093e-05 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248887+0000", 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3009e-05 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248932+0000", 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5445999999999998e-05 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249025+0000", 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3098999999999998e-05 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251503+0000", 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024772629999999999 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252148+0000", 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064507900000000001 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252433+0000", 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028559100000000002 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258304+0000", 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058703100000000001 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258361+0000", 2024-09-13T11:33:05.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7658999999999998e-05 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258407+0000", 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6319999999999997e-05 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.337 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:313 4.2 4:4ba9f189:::smithi20438004-14:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.976549290000001, 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013958247999999999, 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 313 2024-09-13T11:33:05.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249736+0000", 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9999999999999996e-07 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249741+0000", 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5929999999999996e-06 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249750+0000", 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7369999999999995e-06 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249757+0000", 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5360000000000004e-06 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249806+0000", 2024-09-13T11:33:05.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9018999999999999e-05 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249882+0000", 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5943000000000001e-05 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258629+0000", 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087464589999999998 2024-09-13T11:33:05.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259299+0000", 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067041799999999999 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259523+0000", 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223964 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263636+0000", 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041130539999999997 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263669+0000", 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2849999999999999e-05 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263693+0000", 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3774999999999999e-05 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1437 4.2 4:4b088bf8:::smithi20438004-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.422308904000001, 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014642921, 2024-09-13T11:33:05.345 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1437 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803976+0000", 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.35e-06 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803984+0000", 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2080000000000003e-06 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803995+0000", 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804001+0000", 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2870000000000001e-06 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804096+0000", 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5424e-05 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804165+0000", 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8596000000000005e-05 2024-09-13T11:33:05.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.818618+0000", 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452906999999999 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.350 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:05.352 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:05.352 INFO:tasks.rados.rados.0.smithi204.stdout:1225: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:33:05.352 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:05.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:05.353 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:05.350+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:05.385 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:05.452 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:05.451+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:05.496 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:05.496 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:05.496 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:05.496 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:05.505 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:05.516 INFO:tasks.rados.rados.0.smithi204.stdout:1226: finishing set_chunk to oid 30 2024-09-13T11:33:05.517 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 98304 length: 1033728 2024-09-13T11:33:05.517 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-13T11:33:05.517 INFO:tasks.rados.rados.0.smithi204.stdout:1236: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:05.553 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:05.552+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:05.642 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:33:05.648 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:05.648 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:05.648 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:05.648 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:05.649 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:05.654 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:05.652+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:05.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:05.701 INFO:tasks.rados.rados.0.smithi204.stdout:1234: finishing set_chunk to oid 24 2024-09-13T11:33:05.701 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 85 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-09-13T11:33:05.701 INFO:tasks.rados.rados.0.smithi204.stdout:1237: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:05.755 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:05.753+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.387502634000001, 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27446322899999998, 2024-09-13T11:33:05.762 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1100000000000005e-07 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372244+0000", 2024-09-13T11:33:05.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.66e-06 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372256+0000", 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1817000000000001e-05 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372264+0000", 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0220000000000004e-06 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372320+0000", 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6437999999999997e-05 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372329+0000", 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644121+0000", 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27179224899999999 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644271+0000", 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014990000000000001 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644421+0000", 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015035600000000001 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646446+0000", 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00202454 2024-09-13T11:33:05.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646524+0000", 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8839999999999997e-05 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646598+0000", 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3862999999999994e-05 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646654+0000", 2024-09-13T11:33:05.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5609999999999998e-05 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646701+0000", 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6844000000000001e-05 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:207 3.2 3:59a396eb:::smithi20438004-8 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:05.769 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.675105586999997, 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063519737000000007, 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 207 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7099999999999998e-07 2024-09-13T11:33:05.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084636+0000", 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.559e-06 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084640+0000", 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.118e-06 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084644+0000", 2024-09-13T11:33:05.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8640000000000004e-06 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084687+0000", 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2295000000000001e-05 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084780+0000", 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2880000000000005e-05 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.106902+0000", 2024-09-13T11:33:05.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022122822 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112195+0000", 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052927 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.121161+0000", 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089661740000000004 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148052+0000", 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026890595 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148105+0000", 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2673000000000002e-05 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148154+0000", 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9786000000000001e-05 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.775 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:208 3.2 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.675086870999998, 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049409478, 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.776 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 208 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084654+0000", 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084657+0000", 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1769999999999998e-06 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084663+0000", 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8329999999999999e-06 2024-09-13T11:33:05.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084668+0000", 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6200000000000004e-06 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084884+0000", 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021599499999999999 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.085046+0000", 2024-09-13T11:33:05.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016192600000000001 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.094613+0000", 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095668200000000002 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.101900+0000", 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072866160000000001 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.110249+0000", 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008349034 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133903+0000", 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023654272 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133976+0000", 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2581000000000002e-05 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134063+0000", 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7169000000000005e-05 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:211 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.675045842000003, 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047621726000000003, 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.782 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 211 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084695+0000", 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6100000000000002e-07 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084697+0000", 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2419999999999999e-06 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8160000000000001e-06 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084706+0000", 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.313e-06 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086622+0000", 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001916212 2024-09-13T11:33:05.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086720+0000", 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8077000000000003e-05 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100598+0000", 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013877195 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104958+0000", 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043607899999999998 2024-09-13T11:33:05.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115608+0000", 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010650138 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132197+0000", 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016589008999999998 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132257+0000", 2024-09-13T11:33:05.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9948000000000003e-05 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132316+0000", 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8625000000000002e-05 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:212 3.3 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.788 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.675028261999998, 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063179242999999996, 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 212 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7300000000000002e-07 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.255e-06 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084719+0000", 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4800000000000003e-06 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084724+0000", 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.527e-06 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086839+0000", 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021155240000000001 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086947+0000", 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107675 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109128+0000", 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022180947999999999 2024-09-13T11:33:05.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.113055+0000", 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039269029999999998 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.119148+0000", 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060933810000000001 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147786+0000", 2024-09-13T11:33:05.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028638395000000001 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147838+0000", 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1405e-05 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147891+0000", 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3377000000000001e-05 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:216 3.2 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.669089014999997, 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055588580999999998, 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 216 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090653+0000", 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-13T11:33:05.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090663+0000", 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0708e-05 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090675+0000", 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090688+0000", 2024-09-13T11:33:05.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3193000000000001e-05 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096478+0000", 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057898150000000002 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096694+0000", 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021582700000000001 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.103295+0000", 2024-09-13T11:33:05.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006601048 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108559+0000", 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052645449999999998 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.114944+0000", 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063845389999999998 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146121+0000", 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031177449999999999 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146186+0000", 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5235999999999997e-05 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146240+0000", 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3368000000000001e-05 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:220 3.2 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.643913490999999, 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033838506999999997, 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.801 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 220 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115828+0000", 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115832+0000", 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.746e-06 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115842+0000", 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2010000000000005e-06 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115847+0000", 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9740000000000001e-06 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115897+0000", 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0114999999999997e-05 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116103+0000", 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020598799999999999 2024-09-13T11:33:05.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135329+0000", 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019226422999999999 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141061+0000", 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057316709999999998 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.142490+0000", 2024-09-13T11:33:05.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001428805 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149552+0000", 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070625610000000002 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149602+0000", 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9826e-05 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149665+0000", 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3168000000000002e-05 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:226 3.3 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.625362043999999, 2024-09-13T11:33:05.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.062274747999999998, 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 226 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134379+0000", 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1850000000000001e-06 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134392+0000", 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2649999999999999e-05 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134404+0000", 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1615e-05 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134411+0000", 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3509999999999996e-06 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134474+0000", 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2673000000000001e-05 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134662+0000", 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018810100000000001 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149047+0000", 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014384902 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.154919+0000", 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058721379999999998 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.171898+0000", 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016979300999999999 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196539+0000", 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024641252999999998 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196614+0000", 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5160999999999994e-05 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196653+0000", 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8417999999999997e-05 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.813 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:228 3.3 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.619079511999999, 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038699092999999997, 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140662+0000", 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140670+0000", 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8630000000000008e-06 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140696+0000", 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5927000000000001e-05 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140708+0000", 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:33:05.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140804+0000", 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6137999999999993e-05 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140995+0000", 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191316 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.145999+0000", 2024-09-13T11:33:05.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050033129999999997 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152237+0000", 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062383320000000001 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160262+0000", 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080251520000000007 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179164+0000", 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018901823000000002 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179296+0000", 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013243400000000001 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179360+0000", 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3328000000000001e-05 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:229 3.3 3:d73fa370:::smithi20438004-30 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.618945484999998, 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057728689, 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 229 2024-09-13T11:33:05.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140796+0000", 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9599999999999998e-07 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140800+0000", 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140809+0000", 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3669999999999997e-06 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140814+0000", 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141203+0000", 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038959299999999998 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141375+0000", 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017155499999999999 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.156365+0000", 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014989954999999999 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162876+0000", 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065109360000000002 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.187965+0000", 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025088847000000001 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198445+0000", 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010480171 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198487+0000", 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2635000000000001e-05 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198523+0000", 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5941e-05 2024-09-13T11:33:05.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:238 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.60698051, 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.059471032, 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.826 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 238 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152761+0000", 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152772+0000", 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1122e-05 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152782+0000", 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5179999999999993e-06 2024-09-13T11:33:05.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152790+0000", 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7260000000000002e-06 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152856+0000", 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6304999999999996e-05 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153180+0000", 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000323617 2024-09-13T11:33:05.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167201+0000", 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014021264 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.172985+0000", 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057843349999999998 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188702+0000", 2024-09-13T11:33:05.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015716765000000001 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212106+0000", 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023403714999999999 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212169+0000", 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3224000000000003e-05 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212231+0000", 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1841999999999995e-05 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:240 3.2 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.599490533999997, 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034186595, 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.832 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:05.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160251+0000", 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.562e-06 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160258+0000", 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6640000000000001e-06 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160269+0000", 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.152e-05 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160277+0000", 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4939999999999997e-06 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160327+0000", 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0268000000000001e-05 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160497+0000", 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017018900000000001 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.164999+0000", 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045015929999999999 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168431+0000", 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003431698 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.184172+0000", 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015741852000000001 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194334+0000", 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010161646999999999 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194390+0000", 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6218999999999998e-05 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194436+0000", 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5889000000000001e-05 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:242 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 18] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.591102726999999, 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053252359999999999, 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.838 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 242 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168640+0000", 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168652+0000", 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1909000000000001e-05 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168668+0000", 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5888000000000001e-05 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168680+0000", 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.154e-05 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169643+0000", 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096305100000000003 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000139834 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174850+0000", 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005067841 2024-09-13T11:33:05.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.178949+0000", 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040991209999999998 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193069+0000", 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014119166000000001 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221801+0000", 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028732905999999999 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221852+0000", 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0958000000000002e-05 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221890+0000", 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7398999999999999e-05 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:244 3.2 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.579162703000001, 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040499786000000003, 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.845 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:05.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180579+0000", 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.499e-06 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180585+0000", 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4649999999999999e-06 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180598+0000", 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180604+0000", 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5300000000000004e-06 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180669+0000", 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4859999999999999e-05 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180840+0000", 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171791 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.190029+0000", 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091888999999999998 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193312+0000", 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829579999999999 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.207773+0000", 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014461039 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.220989+0000", 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013215644 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221038+0000", 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9421e-05 2024-09-13T11:33:05.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221077+0000", 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8899999999999997e-05 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:247 3.2 3:4f19c0b8:::smithi20438004-48 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.573925799000001, 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040924556000000001, 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.851 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 247 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185816+0000", 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9690000000000001e-06 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185825+0000", 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6980000000000004e-06 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185842+0000", 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6492000000000001e-05 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185852+0000", 2024-09-13T11:33:05.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0412e-05 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185951+0000", 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9312000000000001e-05 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186223+0000", 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027177000000000001 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208403+0000", 2024-09-13T11:33:05.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022179780999999999 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029604150000000001 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.223543+0000", 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012179840000000001 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226659+0000", 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031159989999999999 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226698+0000", 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9020999999999998e-05 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226739+0000", 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0846999999999997e-05 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.857 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:248 3.2 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.573602717999997, 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035109013000000001, 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:05.858 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 248 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186139+0000", 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.333e-06 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186157+0000", 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8488999999999999e-05 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186170+0000", 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2737e-05 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186181+0000", 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1279e-05 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186356+0000", 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174988 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186493+0000", 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137045 2024-09-13T11:33:05.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.203390+0000", 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016897169 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.206673+0000", 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829510000000001 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.214284+0000", 2024-09-13T11:33:05.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076100830000000001 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221141+0000", 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068573740000000003 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221198+0000", 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6916999999999997e-05 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221246+0000", 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8647999999999999e-05 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.10.22:30 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.968565642000002, 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87173941200000005, 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791176+0000", 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:33:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791183+0000", 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.956e-06 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791193+0000", 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8649999999999996e-06 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791200+0000", 2024-09-13T11:33:05.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.148e-06 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791263+0000", 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3038000000000002e-05 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791370+0000", 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010721999999999999 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791704+0000", 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033406100000000003 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.792618+0000", 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091326999999999999 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.793087+0000", 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046985199999999999 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662893+0000", 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86980523300000001 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662914+0000", 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1319e-05 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.869 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.1.22:76 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.954918128999999, 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85814986199999999, 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804824+0000", 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5889999999999999e-06 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804830+0000", 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7139999999999996e-06 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804845+0000", 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5158e-05 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804853+0000", 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9349999999999994e-06 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804910+0000", 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6501e-05 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805017+0000", 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107237 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805213+0000", 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019558100000000001 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806246+0000", 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001033225 2024-09-13T11:33:05.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806260+0000", 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4072e-05 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662951+0000", 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85669131700000001 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662972+0000", 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0533000000000001e-05 2024-09-13T11:33:05.874 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:9 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.914394428000001, 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81776348899999995, 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845348+0000", 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4090000000000001e-06 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845360+0000", 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1420000000000001e-05 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845389+0000", 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9453000000000001e-05 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845404+0000", 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4919000000000001e-05 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845526+0000", 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121588 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845617+0000", 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1119000000000006e-05 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845789+0000", 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017257000000000001 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846396+0000", 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060634700000000003 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846771+0000", 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037541099999999997 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663066+0000", 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81629477399999995 2024-09-13T11:33:05.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663109+0000", 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3479000000000002e-05 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:10 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.910657999000001, 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81407563599999999, 2024-09-13T11:33:05.880 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849083+0000", 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6300000000000004e-07 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849089+0000", 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849101+0000", 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-05 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849106+0000", 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7309999999999998e-06 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849154+0000", 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8077000000000001e-05 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849235+0000", 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0770000000000001e-05 2024-09-13T11:33:05.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849399+0000", 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163385 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850247+0000", 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084889399999999999 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850266+0000", 2024-09-13T11:33:05.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8698e-05 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663146+0000", 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81287977300000003 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663158+0000", 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1928e-05 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:05.885 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:05.886 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:05.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:05.888 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:05.886+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:05.987 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:05.987+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:06.004 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:06.004 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:06.004 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:06.004 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:06.010 INFO:tasks.rados.rados.0.smithi204.stdout:1238: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-13T11:33:06.011 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:06.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:06.070 INFO:tasks.rados.rados.0.smithi204.stdout:1232: got (0) Success 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 0 (ObjNum 225 snap 0 seq_num 225) dirty exists 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1224: done (14 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1225: done (13 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1226: done (12 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1227: done (11 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1228: done (10 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1229: done (9 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1230: done (8 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1232: done (7 left) 2024-09-13T11:33:06.071 INFO:tasks.rados.rados.0.smithi204.stdout:1233: done (6 left) 2024-09-13T11:33:06.072 INFO:tasks.rados.rados.0.smithi204.stdout:1234: done (5 left) 2024-09-13T11:33:06.072 INFO:tasks.rados.rados.0.smithi204.stdout:1235: done (4 left) 2024-09-13T11:33:06.072 INFO:tasks.rados.rados.0.smithi204.stdout:1236: done (3 left) 2024-09-13T11:33:06.072 INFO:tasks.rados.rados.0.smithi204.stdout:1237: done (2 left) 2024-09-13T11:33:06.072 INFO:tasks.rados.rados.0.smithi204.stdout:1238: done (1 left) 2024-09-13T11:33:06.072 INFO:tasks.rados.rados.0.smithi204.stdout:1239: done (0 left) 2024-09-13T11:33:06.072 INFO:tasks.rados.rados.0.smithi204.stdout:1240: delete oid 26 current snap is 0 2024-09-13T11:33:06.073 INFO:tasks.rados.rados.0.smithi204.stdout:1240: done (0 left) 2024-09-13T11:33:06.073 INFO:tasks.rados.rados.0.smithi204.stdout:1241: tier_flush oid 36 2024-09-13T11:33:06.073 INFO:tasks.rados.rados.0.smithi204.stdout:1241: tier_flush oid 36 snap -1 2024-09-13T11:33:06.073 INFO:tasks.rados.rados.0.smithi204.stdout:1242: read oid 3 snap -1 2024-09-13T11:33:06.073 INFO:tasks.rados.rados.0.smithi204.stdout:1242: expect deleted 2024-09-13T11:33:06.073 INFO:tasks.rados.rados.0.smithi204.stdout:1243: tier_promote oid 44 2024-09-13T11:33:06.073 INFO:tasks.rados.rados.0.smithi204.stdout:1244: write (excl) oid 4 current snap is 0 2024-09-13T11:33:06.074 INFO:tasks.rados.rados.0.smithi204.stdout:1244: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-13T11:33:06.078 INFO:tasks.rados.rados.0.smithi204.stdout:1244: writing smithi20438004-4 from 682417 to 1155526 tid 1 2024-09-13T11:33:06.084 INFO:tasks.rados.rados.0.smithi204.stdout:1244: writing smithi20438004-4 from 1567811 to 2169874 tid 2 2024-09-13T11:33:06.088 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:06.086+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:06.092 INFO:tasks.rados.rados.0.smithi204.stdout:1244: writing smithi20438004-4 from 2844123 to 3465605 tid 3 2024-09-13T11:33:06.092 INFO:tasks.rados.rados.0.smithi204.stdout:1245: write oid 45 current snap is 0 2024-09-13T11:33:06.092 INFO:tasks.rados.rados.0.smithi204.stdout:1245: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-13T11:33:06.100 INFO:tasks.rados.rados.0.smithi204.stdout:1245: writing smithi20438004-45 from 699224 to 1247582 tid 1 2024-09-13T11:33:06.107 INFO:tasks.rados.rados.0.smithi204.stdout:1245: writing smithi20438004-45 from 2026293 to 2488367 tid 2 2024-09-13T11:33:06.114 INFO:tasks.rados.rados.0.smithi204.stdout:1245: writing smithi20438004-45 from 3271546 to 3704300 tid 3 2024-09-13T11:33:06.114 INFO:tasks.rados.rados.0.smithi204.stdout:1246: read oid 16 snap -1 2024-09-13T11:33:06.114 INFO:tasks.rados.rados.0.smithi204.stdout:1246: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:33:06.114 INFO:tasks.rados.rados.0.smithi204.stdout:1247: read oid 39 snap -1 2024-09-13T11:33:06.114 INFO:tasks.rados.rados.0.smithi204.stdout:1247: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1248: tier_evict oid 7 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1248: tier_evict oid 7 snap -1 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1249: read oid 12 snap -1 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1249: expect (ObjNum 225 snap 0 seq_num 225) 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1250: read oid 10 snap -1 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1250: expect deleted 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1251: read oid 47 snap -1 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1251: expect deleted 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1252: write (excl) oid 28 current snap is 0 2024-09-13T11:33:06.115 INFO:tasks.rados.rados.0.smithi204.stdout:1252: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-13T11:33:06.122 INFO:tasks.rados.rados.0.smithi204.stdout:1252: writing smithi20438004-28 from 716031 to 1339638 tid 1 2024-09-13T11:33:06.124 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:06.125 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:06.125 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:06.125 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:06.125 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:06.130 INFO:tasks.rados.rados.0.smithi204.stdout:1252: writing smithi20438004-28 from 2084775 to 2690507 tid 2 2024-09-13T11:33:06.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:06.137 INFO:tasks.rados.rados.0.smithi204.stdout:1252: writing smithi20438004-28 from 3298969 to 3916031 tid 3 2024-09-13T11:33:06.138 INFO:tasks.rados.rados.0.smithi204.stdout:1253: read oid 5 snap -1 2024-09-13T11:33:06.138 INFO:tasks.rados.rados.0.smithi204.stdout:1253: expect (ObjNum 181 snap 0 seq_num 181) 2024-09-13T11:33:06.138 INFO:tasks.rados.rados.0.smithi204.stdout:1254: read oid 15 snap -1 2024-09-13T11:33:06.138 INFO:tasks.rados.rados.0.smithi204.stdout:1254: expect deleted 2024-09-13T11:33:06.138 INFO:tasks.rados.rados.0.smithi204.stdout:1255: set_chunk oid 8 target oid 2024-09-13T11:33:06.146 INFO:tasks.rados.rados.0.smithi204.stdout:1255: set_chunk oid 8 offset: 158208 length: 656896 target oid 8415aa8faaa2cd44c06d22f16003447a18c4f4b83781aeb39122c237a6c5d15e offset: 0 snap -1 2024-09-13T11:33:06.146 INFO:tasks.rados.rados.0.smithi204.stdout:1256: read oid 19 snap -1 2024-09-13T11:33:06.146 INFO:tasks.rados.rados.0.smithi204.stdout:1256: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:33:06.146 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:06.146 INFO:tasks.rados.rados.0.smithi204.stdout:1243: got (0) Success 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 46 (ObjNum 223 snap 0 seq_num 223) dirty exists 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: finishing write tid 1 to smithi20438004-4 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: finishing write tid 2 to smithi20438004-4 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1242: done (15 left) 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1243: done (14 left) 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1257: write oid 11 current snap is 0 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: finishing write tid 3 to smithi20438004-4 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: finishing write tid 4 to smithi20438004-4 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: finishing write tid 5 to smithi20438004-4 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: oid 4 updating version 0 to 65 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: oid 4 updating version 65 to 68 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: oid 4 version 68 is already newer than 66 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: oid 4 version 68 is already newer than 67 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 68 (ObjNum 231 snap 0 seq_num 231) dirty exists 2024-09-13T11:33:06.147 INFO:tasks.rados.rados.0.smithi204.stdout:1244: left oid 4 (ObjNum 231 snap 0 seq_num 231) 2024-09-13T11:33:06.148 INFO:tasks.rados.rados.0.smithi204.stdout:1241: got (0) Success 2024-09-13T11:33:06.148 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 222 snap 0 seq_num 222) dirty exists 2024-09-13T11:33:06.148 INFO:tasks.rados.rados.0.smithi204.stdout:1245: finishing write tid 1 to smithi20438004-45 2024-09-13T11:33:06.148 INFO:tasks.rados.rados.0.smithi204.stdout:1257: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-13T11:33:06.154 INFO:tasks.rados.rados.0.smithi204.stdout:1257: writing smithi20438004-11 from 732838 to 1431694 tid 1 2024-09-13T11:33:06.161 INFO:tasks.rados.rados.0.smithi204.stdout:1257: writing smithi20438004-11 from 2026904 to 2492647 tid 2 2024-09-13T11:33:06.168 INFO:tasks.rados.rados.0.smithi204.stdout:1257: writing smithi20438004-11 from 3210039 to 3781690 tid 3 2024-09-13T11:33:06.168 INFO:tasks.rados.rados.0.smithi204.stdout:1241: done (14 left) 2024-09-13T11:33:06.168 INFO:tasks.rados.rados.0.smithi204.stdout:1244: done (13 left) 2024-09-13T11:33:06.168 INFO:tasks.rados.rados.0.smithi204.stdout:1258: set_chunk oid 41 target oid 2024-09-13T11:33:06.177 INFO:tasks.rados.rados.0.smithi204.stdout:1258: set_chunk oid 41 offset: 650240 length: 821760 target oid 0c9709a2429b065738ad4ccf1951cdcd3d90b4e425e56e1c57e8d19140ba7508 offset: 0 snap -1 2024-09-13T11:33:06.177 INFO:tasks.rados.rados.0.smithi204.stdout:1259: read oid 2 snap -1 2024-09-13T11:33:06.177 INFO:tasks.rados.rados.0.smithi204.stdout:1259: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-13T11:33:06.177 INFO:tasks.rados.rados.0.smithi204.stdout:1260: write (excl) oid 48 current snap is 0 2024-09-13T11:33:06.178 INFO:tasks.rados.rados.0.smithi204.stdout:1245: finishing write tid 2 to smithi20438004-45 2024-09-13T11:33:06.178 INFO:tasks.rados.rados.0.smithi204.stdout:1260: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-13T11:33:06.186 INFO:tasks.rados.rados.0.smithi204.stdout:1260: writing smithi20438004-48 from 749645 to 1523750 tid 1 2024-09-13T11:33:06.189 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:06.187+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:06.193 INFO:tasks.rados.rados.0.smithi204.stdout:1260: writing smithi20438004-48 from 2085386 to 2694787 tid 2 2024-09-13T11:33:06.200 INFO:tasks.rados.rados.0.smithi204.stdout:1260: writing smithi20438004-48 from 3237462 to 3949645 tid 3 2024-09-13T11:33:06.200 INFO:tasks.rados.rados.0.smithi204.stdout:1250: done (15 left) 2024-09-13T11:33:06.200 INFO:tasks.rados.rados.0.smithi204.stdout:1251: done (14 left) 2024-09-13T11:33:06.200 INFO:tasks.rados.rados.0.smithi204.stdout:1261: set_chunk oid 25 target oid 2024-09-13T11:33:06.216 INFO:tasks.rados.rados.0.smithi204.stdout:1261: set_chunk oid 25 offset: 987136 length: 1712128 target oid eb54e600ec779be348e40faed86e99703fae4265f0f6aa5b08a284221eca14e2 offset: 0 snap -1 2024-09-13T11:33:06.216 INFO:tasks.rados.rados.0.smithi204.stdout:1262: delete oid 50 current snap is 0 2024-09-13T11:33:06.216 INFO:tasks.rados.rados.0.smithi204.stdout:1248: got (22) Invalid argument 2024-09-13T11:33:06.216 INFO:tasks.rados.rados.0.smithi204.stdout:1246: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:33:06.234 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2019021 2024-09-13T11:33:06.284 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:201 3.1 3:b265d015:::smithi20438004-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.197055771000002, 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030631999, 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 201 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084693+0000", 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4739999999999999e-06 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.884e-06 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:06.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2858e-05 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084748+0000", 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3113e-05 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084807+0000", 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8523999999999997e-05 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084930+0000", 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012317099999999999 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097750+0000", 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012820575000000001 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107462+0000", 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097120990000000001 2024-09-13T11:33:06.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114842+0000", 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073796520000000004 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115192+0000", 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035007400000000002 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115249+0000", 2024-09-13T11:33:06.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7277000000000001e-05 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115323+0000", 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3298000000000002e-05 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:202 3.1 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:06.291 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.196974324000003, 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055756029999999998, 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084773+0000", 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:06.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084777+0000", 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3540000000000002e-06 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084784+0000", 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084789+0000", 2024-09-13T11:33:06.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5480000000000001e-06 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085081+0000", 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000291673 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085197+0000", 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011550800000000001 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109819+0000", 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024622326999999999 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115554+0000", 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057346100000000002 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.139296+0000", 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023742025999999999 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140477+0000", 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011813660000000001 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140507+0000", 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0307e-05 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140528+0000", 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0934e-05 2024-09-13T11:33:06.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:203 3.1 3:970ba768:::smithi20438004-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.196931556000003, 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055588820999999997, 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1900000000000003e-07 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084824+0000", 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3110000000000006e-06 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084831+0000", 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4869999999999996e-06 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084839+0000", 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7840000000000004e-06 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085318+0000", 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047915700000000002 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085404+0000", 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5576999999999997e-05 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.108042+0000", 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022638703999999999 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115550+0000", 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075072150000000002 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132897+0000", 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017347621000000001 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140260+0000", 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073629139999999999 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140331+0000", 2024-09-13T11:33:06.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0994999999999994e-05 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140404+0000", 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2537e-05 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:206 3.1 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:06.304 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.196895935000001, 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022595217000000001, 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:06.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1799999999999995e-07 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084854+0000", 2024-09-13T11:33:06.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1650000000000002e-06 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084861+0000", 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1300000000000003e-06 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084868+0000", 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.55e-06 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085510+0000", 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064168099999999996 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085595+0000", 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5643999999999995e-05 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.091806+0000", 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062107220000000001 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107236+0000", 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015429897999999999 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107281+0000", 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4994000000000003e-05 2024-09-13T11:33:06.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107302+0000", 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1163e-05 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107375+0000", 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2915000000000002e-05 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107446+0000", 2024-09-13T11:33:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0837e-05 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:210 3.1 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.196868799999997, 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047980961000000003, 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.311 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 210 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:06.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084883+0000", 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084890+0000", 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0650000000000001e-06 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084896+0000", 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9000000000000003e-06 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085703+0000", 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000807097 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085822+0000", 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119098 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.104833+0000", 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019011389 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115543+0000", 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01070981 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127458+0000", 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011915189 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132784+0000", 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053251139999999997 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132825+0000", 2024-09-13T11:33:06.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1329999999999999e-05 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132859+0000", 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:215 3.1 3:84f0de12:::smithi20438004-16 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.196842482999998, 2024-09-13T11:33:06.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034172806999999999, 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 215 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0400000000000002e-07 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084907+0000", 2024-09-13T11:33:06.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.666e-06 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084915+0000", 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9330000000000001e-06 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084920+0000", 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8289999999999998e-06 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085942+0000", 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001022029 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.086010+0000", 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8007000000000001e-05 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.100909+0000", 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014898991 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107467+0000", 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065583270000000001 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.117910+0000", 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010443309 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.118971+0000", 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010605510000000001 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119023+0000", 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2114000000000003e-05 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119077+0000", 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3647000000000003e-05 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:218 3.1 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.171159934999999, 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036660768000000003, 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 218 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110588+0000", 2024-09-13T11:33:06.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.761e-06 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110603+0000", 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4542000000000001e-05 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110616+0000", 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2795e-05 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110628+0000", 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1959e-05 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114882+0000", 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042543379999999999 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000143676 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.120883+0000", 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058572549999999996 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.123587+0000", 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027042709999999998 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144002+0000", 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020414505999999999 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147169+0000", 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031675470000000002 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147204+0000", 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4523999999999997e-05 2024-09-13T11:33:06.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147247+0000", 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3594000000000003e-05 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:219 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.170695578999997, 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.054051957999999997, 2024-09-13T11:33:06.330 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 219 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111052+0000", 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.093e-06 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111059+0000", 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9410000000000001e-06 2024-09-13T11:33:06.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111069+0000", 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0519999999999999e-05 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111077+0000", 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8290000000000008e-06 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115373+0000", 2024-09-13T11:33:06.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042952490000000001 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115459+0000", 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6367999999999997e-05 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.129610+0000", 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014151285 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.135388+0000", 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057775270000000002 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149276+0000", 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013888773 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165015+0000", 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015738682 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165065+0000", 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0170000000000002e-05 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165103+0000", 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7521000000000002e-05 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:221 3.1 3:baf08732:::smithi20438004-22 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:06.336 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.165302808, 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034943881000000003, 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 221 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.337 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116445+0000", 2024-09-13T11:33:06.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1680000000000001e-06 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116451+0000", 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0379999999999996e-06 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116461+0000", 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0604e-05 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116469+0000", 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8359999999999992e-06 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116516+0000", 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6332999999999998e-05 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116579+0000", 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3825999999999995e-05 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127789+0000", 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011209869000000001 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132875+0000", 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050855780000000003 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149082+0000", 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016207213000000002 2024-09-13T11:33:06.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151312+0000", 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002230358 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151344+0000", 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1081000000000003e-05 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151388+0000", 2024-09-13T11:33:06.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3977000000000003e-05 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:230 3.1 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.135152335000001, 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046965185999999999, 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.343 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:06.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146596+0000", 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.717e-06 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146606+0000", 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146620+0000", 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4343e-05 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146630+0000", 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3989999999999993e-06 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146705+0000", 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5135999999999993e-05 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146828+0000", 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122714 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165695+0000", 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018866951 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.171485+0000", 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057905220000000002 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185089+0000", 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013604132999999999 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193461+0000", 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083717840000000002 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193507+0000", 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5652000000000003e-05 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193559+0000", 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2624000000000003e-05 2024-09-13T11:33:06.350 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:231 3.1 3:ab075b6a:::smithi20438004-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.134942983999998, 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033558088, 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.351 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 231 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146804+0000", 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3300000000000002e-07 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146808+0000", 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4790000000000001e-06 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146814+0000", 2024-09-13T11:33:06.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3230000000000003e-06 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146817+0000", 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4769999999999999e-06 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146955+0000", 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013720299999999999 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147052+0000", 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7231000000000005e-05 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.156769+0000", 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097173290000000002 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165122+0000", 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083524589999999996 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.173409+0000", 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082871379999999994 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180256+0000", 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006846871 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180302+0000", 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6545999999999998e-05 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180362+0000", 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9499000000000001e-05 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:235 3.1 3:be4a0686:::smithi20438004-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.131944558000001, 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035212407000000001, 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.357 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 235 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149803+0000", 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0580000000000001e-06 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149810+0000", 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149820+0000", 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0464e-05 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149826+0000", 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-13T11:33:06.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149863+0000", 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7416999999999998e-05 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149983+0000", 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011927800000000001 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.160853+0000", 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010870216 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.167925+0000", 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070717159999999996 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180224+0000", 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012298972 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184892+0000", 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046685839999999999 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184959+0000", 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7105000000000002e-05 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185014+0000", 2024-09-13T11:33:06.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5099000000000002e-05 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:236 3.1 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.131830643000001, 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059325392999999997, 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.364 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 236 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:06.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6599999999999996e-07 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149920+0000", 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149929+0000", 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3769999999999996e-06 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149937+0000", 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150123+0000", 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018567200000000001 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150254+0000", 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130426 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181070+0000", 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030816856 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.190793+0000", 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097220870000000008 2024-09-13T11:33:06.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197985+0000", 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007192431 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209174+0000", 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011189536999999999 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209209+0000", 2024-09-13T11:33:06.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4359000000000001e-05 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209241+0000", 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2369000000000001e-05 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:237 3.1 3:9a2730a7:::smithi20438004-38 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.129877856, 2024-09-13T11:33:06.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052738543999999998, 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 237 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151870+0000", 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151875+0000", 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2859999999999999e-06 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151890+0000", 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5557999999999999e-05 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151898+0000", 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7500000000000003e-06 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151934+0000", 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5988000000000002e-05 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151999+0000", 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4394999999999996e-05 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.177811+0000", 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025812831000000001 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185336+0000", 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075243209999999996 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197387+0000", 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012051345999999999 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204491+0000", 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071034999999999996 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204551+0000", 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0003000000000001e-05 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204607+0000", 2024-09-13T11:33:06.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:245 3.1 3:adf25eb4:::smithi20438004-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.100688808000001, 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.03651658, 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.377 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 245 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181059+0000", 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181066+0000", 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5119999999999997e-06 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181078+0000", 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2333e-05 2024-09-13T11:33:06.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181085+0000", 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4449999999999999e-06 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185114+0000", 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040285579999999998 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:33:06.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1935000000000007e-05 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.195343+0000", 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010137147000000001 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200849+0000", 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005505874 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.203835+0000", 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002985705 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217494+0000", 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013658926 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217538+0000", 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4048999999999998e-05 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217574+0000", 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6618000000000001e-05 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:249 3.1 3:81faf980:::smithi20438004-50 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.087770159000002, 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038339094999999997, 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.383 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 249 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193977+0000", 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.11e-06 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193984+0000", 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-13T11:33:06.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0794e-05 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194002+0000", 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.374e-06 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194061+0000", 2024-09-13T11:33:06.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9485999999999999e-05 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194291+0000", 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000229321 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222511+0000", 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028219767999999999 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.227779+0000", 2024-09-13T11:33:06.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052686319999999997 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.231941+0000", 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041617809999999998 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232226+0000", 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028545799999999999 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232277+0000", 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0445000000000002e-05 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232315+0000", 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584999999999997e-05 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:44 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:06.389 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.013346430999999, 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019379767999999999, 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:06.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268402+0000", 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268410+0000", 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2069999999999995e-06 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268426+0000", 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5588999999999999e-05 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268437+0000", 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0773e-05 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270340+0000", 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001903231 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270350+0000", 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0543000000000001e-05 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278988+0000", 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086371350000000006 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279079+0000", 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1012000000000002e-05 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279189+0000", 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110598 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281564+0000", 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023743929999999998 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281584+0000", 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0655000000000001e-05 2024-09-13T11:33:06.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287627+0000", 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060427750000000002 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287716+0000", 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9470999999999998e-05 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287780+0000", 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3329000000000003e-05 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:45 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.013288663000001, 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019566733999999999, 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.396 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6100000000000001e-07 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268462+0000", 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6430000000000001e-06 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268469+0000", 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268475+0000", 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8039999999999998e-06 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270391+0000", 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001916713 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270397+0000", 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8490000000000002e-06 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280010+0000", 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096124780000000007 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280095+0000", 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5506000000000004e-05 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280235+0000", 2024-09-13T11:33:06.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000139405 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281596+0000", 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001361615 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281676+0000", 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0259999999999994e-05 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287969+0000", 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062924000000000001 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287999+0000", 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0270999999999999e-05 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288025+0000", 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5488999999999999e-05 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:26 3.1 3:a6c449f0:::2fa16cfaf91e48313b717fb6e16c839aad29b3524900af3b2b4692c59d034ccb:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.013063293999998, 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019443548000000001, 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:06.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7500000000000001e-07 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268686+0000", 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1270000000000001e-06 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268689+0000", 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8709999999999999e-06 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268694+0000", 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4530000000000004e-06 2024-09-13T11:33:06.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270498+0000", 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00180423 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270502+0000", 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.625e-06 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280882+0000", 2024-09-13T11:33:06.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010380263000000001 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280973+0000", 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1199000000000006e-05 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281088+0000", 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011427800000000001 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282421+0000", 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013333590000000001 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282432+0000", 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.113e-05 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288074+0000", 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056417469999999999 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288104+0000", 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0402000000000001e-05 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288127+0000", 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2589e-05 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.409 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:28 3.1 3:96eef999:::4534b522f789f408dc72053dbc198d7d54ba3a77b9806bd26ef033725ea1a210:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.013047554, 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019504840999999998, 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6300000000000001e-07 2024-09-13T11:33:06.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268702+0000", 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.441e-06 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268709+0000", 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.888e-06 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268714+0000", 2024-09-13T11:33:06.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.519e-06 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270524+0000", 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018093740000000001 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270527+0000", 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.484e-06 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281702+0000", 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011174584 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281828+0000", 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012597699999999999 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281975+0000", 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014744000000000001 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282783+0000", 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080754199999999996 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282862+0000", 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9326999999999995e-05 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288155+0000", 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005293056 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288183+0000", 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8152000000000001e-05 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288204+0000", 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0693999999999999e-05 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.416 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:06.418 INFO:tasks.rados.rados.0.smithi204.stdout:1245: finishing write tid 3 to smithi20438004-45 2024-09-13T11:33:06.418 INFO:tasks.rados.rados.0.smithi204.stdout:1245: finishing write tid 4 to smithi20438004-45 2024-09-13T11:33:06.418 INFO:tasks.rados.rados.0.smithi204.stdout:1245: finishing write tid 5 to smithi20438004-45 2024-09-13T11:33:06.418 INFO:tasks.rados.rados.0.smithi204.stdout:1245: oid 45 updating version 0 to 59 2024-09-13T11:33:06.418 INFO:tasks.rados.rados.0.smithi204.stdout:1245: oid 45 version 59 is already newer than 58 2024-09-13T11:33:06.418 INFO:tasks.rados.rados.0.smithi204.stdout:1245: oid 45 updating version 59 to 60 2024-09-13T11:33:06.419 INFO:tasks.rados.rados.0.smithi204.stdout:1245: oid 45 updating version 60 to 61 2024-09-13T11:33:06.419 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 61 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-09-13T11:33:06.419 INFO:tasks.rados.rados.0.smithi204.stdout:1245: left oid 45 (ObjNum 232 snap 0 seq_num 232) 2024-09-13T11:33:06.419 INFO:tasks.rados.rados.0.smithi204.stdout:1249: expect (ObjNum 225 snap 0 seq_num 225) 2024-09-13T11:33:06.419 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:06.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:06.420 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:06.417+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:06.460 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 167839 2024-09-13T11:33:06.518 INFO:tasks.rados.rados.0.smithi204.stdout:1247: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-13T11:33:06.518 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:06.519 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:06.518+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:06.523 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:06.523 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:06.524 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:06.524 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:06.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:06.593 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:06.619 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:06.618+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:06.631 INFO:tasks.rados.rados.0.smithi204.stdout:1252: finishing write tid 1 to smithi20438004-28 2024-09-13T11:33:06.631 INFO:tasks.rados.rados.0.smithi204.stdout:1252: finishing write tid 2 to smithi20438004-28 2024-09-13T11:33:06.632 INFO:tasks.rados.rados.0.smithi204.stdout:1253: expect (ObjNum 181 snap 0 seq_num 181) 2024-09-13T11:33:06.632 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 133641 2024-09-13T11:33:06.634 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:06.635 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:06.635 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:06.635 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:06.635 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:06.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:06.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:06.714 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 359948 2024-09-13T11:33:06.720 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:06.719+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:06.750 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:06.750 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:06.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:06.750 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:06.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.751 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:254 4.0 4:04aeeab9:::smithi20438004-5:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.751 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:06.752 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.544837407000003, 2024-09-13T11:33:06.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022108323999999999, 2024-09-13T11:33:06.752 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.752 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.752 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.757 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.757 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 254 2024-09-13T11:33:06.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:06.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201594+0000", 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201603+0000", 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.568e-06 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201619+0000", 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6540000000000001e-05 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201629+0000", 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3200000000000006e-06 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201685+0000", 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6078000000000001e-05 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201747+0000", 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2242999999999993e-05 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220103+0000", 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01835548 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220937+0000", 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083429299999999995 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221938+0000", 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010010710000000001 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223567+0000", 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001628846 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223638+0000", 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1165000000000001e-05 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223701+0000", 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3120999999999994e-05 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:268 4.f 4:f653a832:::smithi20438004-19:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.522472, 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01043647, 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 268 2024-09-13T11:33:06.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223958+0000", 2024-09-13T11:33:06.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223960+0000", 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.641e-06 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223968+0000", 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1820000000000006e-06 2024-09-13T11:33:06.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223982+0000", 2024-09-13T11:33:06.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3671e-05 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223989+0000", 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224062+0000", 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2315000000000001e-05 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.224166+0000", 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010401299999999999 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 13,14", 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232681+0000", 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085151910000000001 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.233615+0000", 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00093373600000000001 2024-09-13T11:33:06.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234090+0000", 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000475699 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234332+0000", 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241895 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234362+0000", 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9889000000000001e-05 2024-09-13T11:33:06.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234395+0000", 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2471000000000001e-05 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:285 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.511852947999998, 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012191677, 2024-09-13T11:33:06.771 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 285 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234578+0000", 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234583+0000", 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9110000000000001e-06 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234593+0000", 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234598+0000", 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9509999999999997e-06 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234639+0000", 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1826999999999998e-05 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234688+0000", 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9039000000000003e-05 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239203+0000", 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045143739999999998 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239979+0000", 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077664100000000001 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.242634+0000", 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026544899999999998 2024-09-13T11:33:06.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246683+0000", 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040491499999999996 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246723+0000", 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0429000000000002e-05 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246769+0000", 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5430999999999999e-05 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1715 4.f 4:fea8c5b2:::smithi20438004-39:head [write 769787~704463,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.478044008000001, 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020210576000000001, 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1715 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268388+0000", 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269172+0000", 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078478 2024-09-13T11:33:06.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269184+0000", 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.182e-05 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269194+0000", 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8570000000000007e-06 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269269+0000", 2024-09-13T11:33:06.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4795000000000005e-05 2024-09-13T11:33:06.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269283+0000", 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275426+0000", 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061428050000000003 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275540+0000", 2024-09-13T11:33:06.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011432199999999999 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275658+0000", 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118071 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278252+0000", 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594224 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279142+0000", 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088964600000000001 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288321+0000", 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091785600000000005 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288363+0000", 2024-09-13T11:33:06.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2957000000000002e-05 2024-09-13T11:33:06.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288492+0000", 2024-09-13T11:33:06.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012896899999999999 2024-09-13T11:33:06.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288597+0000", 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010429000000000001 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1716 4.f 4:fea8c5b2:::smithi20438004-39:head [write 2267961~101826] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:06.789 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.477217151000001, 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019610079999999998, 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1716 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.790 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269214+0000", 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9699999999999995e-07 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269305+0000", 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1330999999999997e-05 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269314+0000", 2024-09-13T11:33:06.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9209999999999998e-06 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269321+0000", 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269363+0000", 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2388999999999997e-05 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269372+0000", 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277175+0000", 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078032580000000004 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277247+0000", 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1773999999999994e-05 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277354+0000", 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010704000000000001 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288308+0000", 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010953353000000001 2024-09-13T11:33:06.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288328+0000", 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0353e-05 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288640+0000", 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031178999999999999 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288706+0000", 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6749999999999996e-05 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288765+0000", 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8876999999999999e-05 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288823+0000", 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7806999999999997e-05 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1717 4.f 4:fea8c5b2:::smithi20438004-39:head [setxattr _header (56) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:06.798 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.477092889000001, 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019744788999999999, 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1717 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.799 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269338+0000", 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269342+0000", 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.924e-06 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269349+0000", 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9619999999999996e-06 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269355+0000", 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1210000000000003e-06 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269399+0000", 2024-09-13T11:33:06.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4360000000000002e-05 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5570000000000001e-06 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278293+0000", 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088867549999999997 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278372+0000", 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9561999999999995e-05 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278492+0000", 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119654 2024-09-13T11:33:06.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288335+0000", 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098426840000000008 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288672+0000", 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033785899999999998 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288863+0000", 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019105599999999999 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288904+0000", 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0738000000000002e-05 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288949+0000", 2024-09-13T11:33:06.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4443000000000001e-05 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289082+0000", 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013346199999999999 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.807 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1718 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.477063415, 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019946973, 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1718 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:06.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:06.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8999999999999996e-07 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269373+0000", 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8629999999999996e-06 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269379+0000", 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4979999999999997e-06 2024-09-13T11:33:06.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269383+0000", 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7260000000000002e-06 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269443+0000", 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9616e-05 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269451+0000", 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279055+0000", 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009604586 2024-09-13T11:33:06.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279104+0000", 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8131999999999999e-05 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289119+0000", 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010015616 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289177+0000", 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8060999999999999e-05 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289314+0000", 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000136525 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1732 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.477038919999998, 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017069699000000001, 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.816 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1732 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269392+0000", 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269395+0000", 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-06 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269400+0000", 2024-09-13T11:33:06.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.198e-06 2024-09-13T11:33:06.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:06.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9410000000000004e-06 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269514+0000", 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108124 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269524+0000", 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:33:06.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279170+0000", 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096457209999999995 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279248+0000", 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8147999999999998e-05 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286461+0000", 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072129789999999996 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1733 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.476995943999999, 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0187283, 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1733 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.823 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269435+0000", 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7800000000000001e-07 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269438+0000", 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.535e-06 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269447+0000", 2024-09-13T11:33:06.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6559999999999998e-06 2024-09-13T11:33:06.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269452+0000", 2024-09-13T11:33:06.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.972e-06 2024-09-13T11:33:06.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269552+0000", 2024-09-13T11:33:06.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9820000000000003e-05 2024-09-13T11:33:06.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269556+0000", 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286496+0000", 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016940182000000002 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286550+0000", 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3640999999999997e-05 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288163+0000", 2024-09-13T11:33:06.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016126720000000001 2024-09-13T11:33:06.830 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.831 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.831 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1734 4.f 4:f653a832:::smithi20438004-19:head [read 1042153~2571319,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.476930042999999, 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021098286000000001, 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.833 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1734 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:06.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269501+0000", 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269504+0000", 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.58e-06 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269511+0000", 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6200000000000001e-06 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269517+0000", 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7030000000000003e-06 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269580+0000", 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269583+0000", 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5020000000000001e-06 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289341+0000", 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019757429 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289385+0000", 2024-09-13T11:33:06.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3939000000000001e-05 2024-09-13T11:33:06.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.290598+0000", 2024-09-13T11:33:06.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012136130000000001 2024-09-13T11:33:06.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.838 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.839 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.839 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.12.0:167 4.6 e51/31 4:6978b14d:::smithi20438004-23:head v 51'52, mlcod=45'50)", 2024-09-13T11:33:06.839 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:06.839 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.554815699999999, 2024-09-13T11:33:06.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013003298, 2024-09-13T11:33:06.840 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.840 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.840 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.840 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191616+0000", 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0270000000000001e-06 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196708+0000", 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050925720000000001 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196728+0000", 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9616999999999999e-05 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196738+0000", 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8330000000000006e-06 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196817+0000", 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9225000000000001e-05 2024-09-13T11:33:06.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196841+0000", 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3972e-05 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204573+0000", 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077323629999999999 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204618+0000", 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4688999999999999e-05 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1770 4.0 4:001d90c2:::smithi20438004-28:head [read 0~2218524,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:06.845 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.476641082, 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011933050000000001, 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1770 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269791+0000", 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269795+0000", 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269803+0000", 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269810+0000", 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.246e-06 2024-09-13T11:33:06.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273037+0000", 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032276700000000002 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273048+0000", 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0511e-05 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.277997+0000", 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049497539999999998 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278150+0000", 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152406 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281722+0000", 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00357224 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.851 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.852 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1771 4.0 4:001d90c2:::smithi20438004-28:head [read 633380~1585144,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:06.852 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:06.852 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.476611078000001, 2024-09-13T11:33:06.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013022562, 2024-09-13T11:33:06.852 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1771 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269820+0000", 2024-09-13T11:33:06.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4000000000000002e-07 2024-09-13T11:33:06.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269821+0000", 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9269999999999999e-06 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269825+0000", 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8789999999999996e-06 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269829+0000", 2024-09-13T11:33:06.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4740000000000001e-06 2024-09-13T11:33:06.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273144+0000", 2024-09-13T11:33:06.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033155459999999999 2024-09-13T11:33:06.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273155+0000", 2024-09-13T11:33:06.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.062e-05 2024-09-13T11:33:06.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282280+0000", 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091252620000000003 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282403+0000", 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012240499999999999 2024-09-13T11:33:06.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282842+0000", 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043900900000000002 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1772 4.0 4:001d90c2:::smithi20438004-28:head [sparse-read 2038646~179878] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.476597633000001, 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013177762000000001, 2024-09-13T11:33:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1772 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9499999999999998e-07 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269834+0000", 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:33:06.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269837+0000", 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.005e-06 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269843+0000", 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6389999999999997e-06 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273188+0000", 2024-09-13T11:33:06.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033454840000000001 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273196+0000", 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0830000000000004e-06 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282874+0000", 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096776810000000005 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282924+0000", 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0095000000000001e-05 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.283010+0000", 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6216000000000003e-05 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2324 4.16 4:6978b14d:::smithi20438004-23:head [write 544593~799729,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:06.865 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.609349652000001, 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14427186, 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2324 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7100000000000001e-07 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137660+0000", 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057879000000000003 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137671+0000", 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0518e-05 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137675+0000", 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137754+0000", 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8492000000000007e-05 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137765+0000", 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1027e-05 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273433+0000", 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13566771999999999 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273569+0000", 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013616999999999999 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273699+0000", 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130142 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281227+0000", 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075284699999999998 2024-09-13T11:33:06.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281259+0000", 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1610999999999999e-05 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281298+0000", 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9138999999999999e-05 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281352+0000", 2024-09-13T11:33:06.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2325 4.16 4:6978b14d:::smithi20438004-23:head [write 1761799~776065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.602684335999999, 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13997515799999999, 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.872 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2325 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143748+0000", 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000001e-06 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144385+0000", 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000637125 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144396+0000", 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0633e-05 2024-09-13T11:33:06.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144401+0000", 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2469999999999999e-06 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144457+0000", 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5566999999999998e-05 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144467+0000", 2024-09-13T11:33:06.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8819999999999996e-06 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275371+0000", 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13090408000000001 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275440+0000", 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8749000000000002e-05 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275530+0000", 2024-09-13T11:33:06.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0236999999999997e-05 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282215+0000", 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066849240000000001 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283575+0000", 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001360775 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283629+0000", 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283721+0000", 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1799999999999995e-05 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2326 4.16 4:6978b14d:::smithi20438004-23:head [write 2948352~396241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.598318915, 2024-09-13T11:33:06.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13583619799999999, 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2326 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:06.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148113+0000", 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148305+0000", 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000192008 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148316+0000", 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0293e-05 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148327+0000", 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1489e-05 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148398+0000", 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0421999999999998e-05 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148407+0000", 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1090000000000004e-06 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279356+0000", 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13094884700000001 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279444+0000", 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279534+0000", 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9573000000000005e-05 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282581+0000", 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030473990000000001 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283856+0000", 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001275307 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283901+0000", 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5238999999999998e-05 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283947+0000", 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5972e-05 2024-09-13T11:33:06.884 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2327 4.16 4:6978b14d:::smithi20438004-23:head [setxattr _header (56) in=7b,truncate 3344593] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.598091415000001, 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13580746099999999, 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.885 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2327 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.34e-07 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148341+0000", 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0999999999999998e-06 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148363+0000", 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2045999999999999e-05 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148369+0000", 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.665e-06 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148463+0000", 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4475999999999995e-05 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148474+0000", 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0319999999999999e-05 2024-09-13T11:33:06.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280931+0000", 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132457142 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280994+0000", 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2693000000000004e-05 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281088+0000", 2024-09-13T11:33:06.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3993000000000006e-05 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282632+0000", 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001544199 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283989+0000", 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013571670000000001 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284048+0000", 2024-09-13T11:33:06.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9070999999999997e-05 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284146+0000", 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8255e-05 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2328 4.16 4:6978b14d:::smithi20438004-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.5980569, 2024-09-13T11:33:06.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13592306100000001, 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2328 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148374+0000", 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6899999999999999e-07 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148375+0000", 2024-09-13T11:33:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148378+0000", 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0989999999999999e-06 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148381+0000", 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148500+0000", 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119052 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148509+0000", 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8759999999999994e-06 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281377+0000", 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132868244 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281422+0000", 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4909e-05 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284171+0000", 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027497300000000001 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284216+0000", 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4626999999999998e-05 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284296+0000", 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0275e-05 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2687 4.10 4:0c9026a3:::smithi20438004-36:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.67460471600000005, 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029783952999999998, 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:06.897 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2687 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:06.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071827+0000", 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.096e-06 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071831+0000", 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8210000000000001e-06 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:06.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071843+0000", 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1715e-05 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071848+0000", 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071899+0000", 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0290000000000001e-05 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071988+0000", 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.101609+0000", 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0296212 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:06.901 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:06.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:06.904 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:06.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:06.906 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:06.907 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:06.907 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:06.908 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:06.908 INFO:tasks.rados.rados.0.smithi204.stdout:1252: finishing write tid 3 to smithi20438004-28 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1252: finishing write tid 4 to smithi20438004-28 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1245: done (15 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1246: done (14 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1247: done (13 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1248: done (12 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1249: done (11 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1253: done (10 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1254: done (9 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1262: done (8 left) 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1263: read oid 14 snap -1 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1263: expect deleted 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1264: set_chunk oid 15 target oid 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1264: set_chunk oid 15 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:06.909 INFO:tasks.rados.rados.0.smithi204.stdout:1265: set_chunk oid 39 target oid 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1265: set_chunk oid 39 offset: 762368 length: 2271744 target oid d046d2069df6e2b25fde3fe5e366fcff0435611cd9d893fbb5f4f2b37b02f422 offset: 0 snap -1 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1266: tier_flush oid 46 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1266: tier_flush oid 46 snap -1 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1252: finishing write tid 5 to smithi20438004-28 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1252: oid 28 updating version 0 to 58 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1252: oid 28 updating version 58 to 61 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1252: oid 28 version 61 is already newer than 60 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1252: oid 28 version 61 is already newer than 59 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 61 (ObjNum 233 snap 0 seq_num 233) dirty exists 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1252: left oid 28 (ObjNum 233 snap 0 seq_num 233) 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1255: finishing set_chunk to oid 8 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 28 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:1256: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:33:06.910 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 508656 2024-09-13T11:33:06.911 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:06.911 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:06.901+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:06.911 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:06.912 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:06.912 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:06.915 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:06.915 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:06.933 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:06.933 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:06.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:06.950 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:06.950 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:06.967 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:06.967 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:06.984 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:06.984 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:07.001 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:07.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:07.003 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:07.002+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:07.005 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:07.006 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 318168 2024-09-13T11:33:07.015 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.015 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: finishing write tid 1 to smithi20438004-11 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: finishing write tid 2 to smithi20438004-11 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: finishing write tid 3 to smithi20438004-11 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: finishing write tid 4 to smithi20438004-11 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: finishing write tid 5 to smithi20438004-11 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: oid 11 updating version 0 to 109 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: oid 11 updating version 109 to 110 2024-09-13T11:33:07.057 INFO:tasks.rados.rados.0.smithi204.stdout:1257: oid 11 version 110 is already newer than 108 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1257: oid 11 version 110 is already newer than 107 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 110 (ObjNum 234 snap 0 seq_num 234) dirty exists 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1257: left oid 11 (ObjNum 234 snap 0 seq_num 234) 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1252: done (11 left) 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1255: done (10 left) 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1256: done (9 left) 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1257: done (8 left) 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1267: write (excl) oid 11 current snap is 0 2024-09-13T11:33:07.058 INFO:tasks.rados.rados.0.smithi204.stdout:1267: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-13T11:33:07.064 INFO:tasks.rados.rados.0.smithi204.stdout:1267: writing smithi20438004-11 from 783259 to 1424215 tid 1 2024-09-13T11:33:07.070 INFO:tasks.rados.rados.0.smithi204.stdout:1267: writing smithi20438004-11 from 1918703 to 2531773 tid 2 2024-09-13T11:33:07.075 INFO:tasks.rados.rados.0.smithi204.stdout:1267: writing smithi20438004-11 from 3008661 to 3414128 tid 3 2024-09-13T11:33:07.079 INFO:tasks.rados.rados.0.smithi204.stdout:1267: writing smithi20438004-11 from 3821939 to 3983259 tid 4 2024-09-13T11:33:07.079 INFO:tasks.rados.rados.0.smithi204.stdout:1268: set_chunk oid 20 target oid 2024-09-13T11:33:07.083 INFO:tasks.rados.rados.0.smithi204.stdout:1268: set_chunk oid 20 offset: 1631232 length: 577024 target oid ff0982414f005b075194cdd79dfe282c01e249b1ba9dc4cad5cf08d81d05b560 offset: 0 snap -1 2024-09-13T11:33:07.083 INFO:tasks.rados.rados.0.smithi204.stdout:1269: set_chunk oid 21 target oid 2024-09-13T11:33:07.091 INFO:tasks.rados.rados.0.smithi204.stdout:1269: set_chunk oid 21 offset: 2664448 length: 1002496 target oid 1766050513f03e0837803603279f6d6a073769d9f9687c8dd2dac70b97452801 offset: 0 snap -1 2024-09-13T11:33:07.091 INFO:tasks.rados.rados.0.smithi204.stdout:1270: write oid 45 current snap is 0 2024-09-13T11:33:07.091 INFO:tasks.rados.rados.0.smithi204.stdout:1258: finishing set_chunk to oid 41 2024-09-13T11:33:07.092 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 85 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-13T11:33:07.092 INFO:tasks.rados.rados.0.smithi204.stdout:1259: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-13T11:33:07.092 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:07.104 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:07.103+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:07.128 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.128 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:07.152 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:07.206 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:07.204+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:07.212 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:07.241 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.241 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.250 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:07.275 INFO:tasks.rados.rados.0.smithi204.stdout:1260: finishing write tid 1 to smithi20438004-48 2024-09-13T11:33:07.275 INFO:tasks.rados.rados.0.smithi204.stdout:1260: finishing write tid 2 to smithi20438004-48 2024-09-13T11:33:07.276 INFO:tasks.rados.rados.0.smithi204.stdout:1260: finishing write tid 3 to smithi20438004-48 2024-09-13T11:33:07.276 INFO:tasks.rados.rados.0.smithi204.stdout:1270: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-13T11:33:07.281 INFO:tasks.rados.rados.0.smithi204.stdout:1270: writing smithi20438004-45 from 400066 to 1116271 tid 1 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1270: writing smithi20438004-45 from 1860832 to 2000066 tid 2 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1258: done (11 left) 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1259: done (10 left) 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1271: tier_promote oid 47 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1260: finishing write tid 4 to smithi20438004-48 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1260: finishing write tid 5 to smithi20438004-48 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1260: oid 48 updating version 0 to 61 2024-09-13T11:33:07.289 INFO:tasks.rados.rados.0.smithi204.stdout:1260: oid 48 updating version 61 to 62 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1260: oid 48 version 62 is already newer than 59 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1260: oid 48 version 62 is already newer than 60 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 62 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1260: left oid 48 (ObjNum 235 snap 0 seq_num 235) 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1261: finishing set_chunk to oid 25 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 76 (ObjNum 193 snap 0 seq_num 193) dirty exists 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1264: finishing set_chunk to oid 15 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1264: got expected ENOENT (src dne) 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1265: finishing set_chunk to oid 39 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 762368 length: 2271744 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1266: got (0) Success 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-09-13T11:33:07.290 INFO:tasks.rados.rados.0.smithi204.stdout:1260: done (10 left) 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1261: done (9 left) 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1263: done (8 left) 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1264: done (7 left) 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1265: done (6 left) 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1266: done (5 left) 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1272: read oid 23 snap -1 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1272: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1273: tier_promote oid 10 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1267: finishing write tid 1 to smithi20438004-11 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1267: finishing write tid 2 to smithi20438004-11 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1267: finishing write tid 3 to smithi20438004-11 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1267: finishing write tid 4 to smithi20438004-11 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1267: finishing write tid 5 to smithi20438004-11 2024-09-13T11:33:07.291 INFO:tasks.rados.rados.0.smithi204.stdout:1267: finishing write tid 6 to smithi20438004-11 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1267: oid 11 updating version 0 to 114 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1267: oid 11 version 114 is already newer than 111 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1267: oid 11 version 114 is already newer than 112 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1267: oid 11 version 114 is already newer than 113 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1267: oid 11 updating version 114 to 115 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 115 (ObjNum 237 snap 0 seq_num 237) dirty exists 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1267: left oid 11 (ObjNum 237 snap 0 seq_num 237) 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1268: finishing set_chunk to oid 20 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 49 (ObjNum 183 snap 0 seq_num 183) dirty exists 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1267: done (6 left) 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1268: done (5 left) 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1274: read oid 34 snap -1 2024-09-13T11:33:07.292 INFO:tasks.rados.rados.0.smithi204.stdout:1274: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:07.293 INFO:tasks.rados.rados.0.smithi204.stdout:1275: read oid 24 snap -1 2024-09-13T11:33:07.293 INFO:tasks.rados.rados.0.smithi204.stdout:1275: expect (ObjNum 182 snap 0 seq_num 182) 2024-09-13T11:33:07.293 INFO:tasks.rados.rados.0.smithi204.stdout:1276: write oid 32 current snap is 0 2024-09-13T11:33:07.293 INFO:tasks.rados.rados.0.smithi204.stdout:1276: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-13T11:33:07.294 INFO:tasks.rados.rados.0.smithi204.stdout:1276: writing smithi20438004-32 from 416873 to 1208327 tid 1 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:1276: writing smithi20438004-32 from 1919314 to 2016873 tid 2 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:1277: read oid 37 snap -1 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:1277: expect deleted 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:1278: write oid 5 current snap is 0 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:1269: finishing set_chunk to oid 21 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 31 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:1270: finishing write tid 1 to smithi20438004-45 2024-09-13T11:33:07.300 INFO:tasks.rados.rados.0.smithi204.stdout:1278: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-13T11:33:07.305 INFO:tasks.rados.rados.0.smithi204.stdout:1278: writing smithi20438004-5 from 433680 to 900383 tid 1 2024-09-13T11:33:07.306 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:07.305+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:07.311 INFO:tasks.rados.rados.0.smithi204.stdout:1278: writing smithi20438004-5 from 1577796 to 2033680 tid 2 2024-09-13T11:33:07.311 INFO:tasks.rados.rados.0.smithi204.stdout:1269: done (9 left) 2024-09-13T11:33:07.311 INFO:tasks.rados.rados.0.smithi204.stdout:1279: set_chunk oid 36 target oid 2024-09-13T11:33:07.327 INFO:tasks.rados.rados.0.smithi204.stdout:1279: set_chunk oid 36 offset: 751104 length: 1522176 target oid 8cfcc1a0f0874c4fbfefeff728cc9f639658e6bccfd6ef91bedbd8f05fb4142c offset: 0 snap -1 2024-09-13T11:33:07.327 INFO:tasks.rados.rados.0.smithi204.stdout:1280: read oid 15 snap -1 2024-09-13T11:33:07.328 INFO:tasks.rados.rados.0.smithi204.stdout:1280: expect deleted 2024-09-13T11:33:07.328 INFO:tasks.rados.rados.0.smithi204.stdout:1281: write oid 1 current snap is 0 2024-09-13T11:33:07.328 INFO:tasks.rados.rados.0.smithi204.stdout:1271: got (2) No such file or directory 2024-09-13T11:33:07.328 INFO:tasks.rados.rados.0.smithi204.stdout:1271: got expected ENOENT (src dne) 2024-09-13T11:33:07.328 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 41 (ObjNum -513827616 snap 32547 seq_num 304) clean dne 2024-09-13T11:33:07.328 INFO:tasks.rados.rados.0.smithi204.stdout:1270: finishing write tid 2 to smithi20438004-45 2024-09-13T11:33:07.329 INFO:tasks.rados.rados.0.smithi204.stdout:1270: finishing write tid 3 to smithi20438004-45 2024-09-13T11:33:07.329 INFO:tasks.rados.rados.0.smithi204.stdout:1270: finishing write tid 4 to smithi20438004-45 2024-09-13T11:33:07.329 INFO:tasks.rados.rados.0.smithi204.stdout:1270: oid 45 updating version 0 to 64 2024-09-13T11:33:07.329 INFO:tasks.rados.rados.0.smithi204.stdout:1270: oid 45 version 64 is already newer than 63 2024-09-13T11:33:07.329 INFO:tasks.rados.rados.0.smithi204.stdout:1270: oid 45 version 64 is already newer than 62 2024-09-13T11:33:07.329 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 64 (ObjNum 238 snap 0 seq_num 238) dirty exists 2024-09-13T11:33:07.329 INFO:tasks.rados.rados.0.smithi204.stdout:1270: left oid 45 (ObjNum 238 snap 0 seq_num 238) 2024-09-13T11:33:07.330 INFO:tasks.rados.rados.0.smithi204.stdout:1281: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-13T11:33:07.331 INFO:tasks.rados.rados.0.smithi204.stdout:1281: writing smithi20438004-1 from 450487 to 992439 tid 1 2024-09-13T11:33:07.338 INFO:tasks.rados.rados.0.smithi204.stdout:1281: writing smithi20438004-1 from 1636278 to 2050487 tid 2 2024-09-13T11:33:07.338 INFO:tasks.rados.rados.0.smithi204.stdout:1270: done (11 left) 2024-09-13T11:33:07.338 INFO:tasks.rados.rados.0.smithi204.stdout:1271: done (10 left) 2024-09-13T11:33:07.338 INFO:tasks.rados.rados.0.smithi204.stdout:1282: read oid 49 snap -1 2024-09-13T11:33:07.339 INFO:tasks.rados.rados.0.smithi204.stdout:1282: expect deleted 2024-09-13T11:33:07.339 INFO:tasks.rados.rados.0.smithi204.stdout:1283: write (excl) oid 27 current snap is 0 2024-09-13T11:33:07.339 INFO:tasks.rados.rados.0.smithi204.stdout:1283: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-13T11:33:07.343 INFO:tasks.rados.rados.0.smithi204.stdout:1283: writing smithi20438004-27 from 467294 to 1084495 tid 1 2024-09-13T11:33:07.349 INFO:tasks.rados.rados.0.smithi204.stdout:1283: writing smithi20438004-27 from 1578407 to 2067294 tid 2 2024-09-13T11:33:07.349 INFO:tasks.rados.rados.0.smithi204.stdout:1284: read oid 16 snap -1 2024-09-13T11:33:07.349 INFO:tasks.rados.rados.0.smithi204.stdout:1284: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1285: read oid 20 snap -1 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1285: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1286: read oid 21 snap -1 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1286: expect (ObjNum 226 snap 0 seq_num 226) 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1287: read oid 7 snap -1 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1287: expect (ObjNum 197 snap 0 seq_num 197) 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1273: got (2) No such file or directory 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1273: got expected ENOENT (src dne) 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 110 (ObjNum 129 snap 0 seq_num 3594710168) clean dne 2024-09-13T11:33:07.350 INFO:tasks.rados.rados.0.smithi204.stdout:1274: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:07.363 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1270895 2024-09-13T11:33:07.374 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.374 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.378 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:07.407 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:07.406+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:07.474 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3164312 2024-09-13T11:33:07.485 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 265301 2024-09-13T11:33:07.507 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:07.507+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:07.522 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.522 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.532 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:07.563 INFO:tasks.rados.rados.0.smithi204.stdout:1272: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:07.608 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:07.607+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:07.636 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.637 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.645 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:07.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:33:07.709 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:07.708+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:07.785 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:07.791 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.791 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:07.809 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:07.809+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:07.901 INFO:tasks.rados.rados.0.smithi204.stdout:1276: finishing write tid 1 to smithi20438004-32 2024-09-13T11:33:07.901 INFO:tasks.rados.rados.0.smithi204.stdout:1276: finishing write tid 2 to smithi20438004-32 2024-09-13T11:33:07.901 INFO:tasks.rados.rados.0.smithi204.stdout:1276: finishing write tid 3 to smithi20438004-32 2024-09-13T11:33:07.901 INFO:tasks.rados.rados.0.smithi204.stdout:1276: finishing write tid 4 to smithi20438004-32 2024-09-13T11:33:07.901 INFO:tasks.rados.rados.0.smithi204.stdout:1276: oid 32 updating version 0 to 48 2024-09-13T11:33:07.901 INFO:tasks.rados.rados.0.smithi204.stdout:1276: oid 32 updating version 48 to 50 2024-09-13T11:33:07.901 INFO:tasks.rados.rados.0.smithi204.stdout:1276: oid 32 version 50 is already newer than 49 2024-09-13T11:33:07.902 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 50 (ObjNum 239 snap 0 seq_num 239) dirty exists 2024-09-13T11:33:07.902 INFO:tasks.rados.rados.0.smithi204.stdout:1276: left oid 32 (ObjNum 239 snap 0 seq_num 239) 2024-09-13T11:33:07.902 INFO:tasks.rados.rados.0.smithi204.stdout:1275: expect (ObjNum 182 snap 0 seq_num 182) 2024-09-13T11:33:07.904 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:07.904 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:07.910 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:07.909+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:07.913 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:08.011 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:08.010+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:08.018 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.018 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.027 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:08.082 INFO:tasks.rados.rados.0.smithi204.stdout:1278: finishing write tid 1 to smithi20438004-5 2024-09-13T11:33:08.082 INFO:tasks.rados.rados.0.smithi204.stdout:1278: finishing write tid 2 to smithi20438004-5 2024-09-13T11:33:08.082 INFO:tasks.rados.rados.0.smithi204.stdout:1278: finishing write tid 3 to smithi20438004-5 2024-09-13T11:33:08.083 INFO:tasks.rados.rados.0.smithi204.stdout:1278: finishing write tid 4 to smithi20438004-5 2024-09-13T11:33:08.083 INFO:tasks.rados.rados.0.smithi204.stdout:1278: oid 5 updating version 0 to 63 2024-09-13T11:33:08.083 INFO:tasks.rados.rados.0.smithi204.stdout:1278: oid 5 version 63 is already newer than 62 2024-09-13T11:33:08.083 INFO:tasks.rados.rados.0.smithi204.stdout:1278: oid 5 updating version 63 to 64 2024-09-13T11:33:08.083 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 64 (ObjNum 240 snap 0 seq_num 240) dirty exists 2024-09-13T11:33:08.084 INFO:tasks.rados.rados.0.smithi204.stdout:1278: left oid 5 (ObjNum 240 snap 0 seq_num 240) 2024-09-13T11:33:08.084 INFO:tasks.rados.rados.0.smithi204.stdout:1279: finishing set_chunk to oid 36 2024-09-13T11:33:08.084 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 36 set_chunk returned error code -95 offset: 751104 length: 1522176 2024-09-13T11:33:08.084 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 0 (ObjNum 222 snap 0 seq_num 222) dirty exists 2024-09-13T11:33:08.084 INFO:tasks.rados.rados.0.smithi204.stdout:1272: done (15 left) 2024-09-13T11:33:08.084 INFO:tasks.rados.rados.0.smithi204.stdout:1273: done (14 left) 2024-09-13T11:33:08.085 INFO:tasks.rados.rados.0.smithi204.stdout:1274: done (13 left) 2024-09-13T11:33:08.085 INFO:tasks.rados.rados.0.smithi204.stdout:1275: done (12 left) 2024-09-13T11:33:08.085 INFO:tasks.rados.rados.0.smithi204.stdout:1276: done (11 left) 2024-09-13T11:33:08.085 INFO:tasks.rados.rados.0.smithi204.stdout:1277: done (10 left) 2024-09-13T11:33:08.085 INFO:tasks.rados.rados.0.smithi204.stdout:1278: done (9 left) 2024-09-13T11:33:08.086 INFO:tasks.rados.rados.0.smithi204.stdout:1279: done (8 left) 2024-09-13T11:33:08.086 INFO:tasks.rados.rados.0.smithi204.stdout:1288: read oid 48 snap -1 2024-09-13T11:33:08.086 INFO:tasks.rados.rados.0.smithi204.stdout:1288: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-13T11:33:08.086 INFO:tasks.rados.rados.0.smithi204.stdout:1289: read oid 23 snap -1 2024-09-13T11:33:08.087 INFO:tasks.rados.rados.0.smithi204.stdout:1289: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:08.087 INFO:tasks.rados.rados.0.smithi204.stdout:1290: read oid 47 snap -1 2024-09-13T11:33:08.087 INFO:tasks.rados.rados.0.smithi204.stdout:1290: expect deleted 2024-09-13T11:33:08.087 INFO:tasks.rados.rados.0.smithi204.stdout:1291: read oid 4 snap -1 2024-09-13T11:33:08.087 INFO:tasks.rados.rados.0.smithi204.stdout:1291: expect (ObjNum 231 snap 0 seq_num 231) 2024-09-13T11:33:08.087 INFO:tasks.rados.rados.0.smithi204.stdout:1292: tier_promote oid 50 2024-09-13T11:33:08.088 INFO:tasks.rados.rados.0.smithi204.stdout:1281: finishing write tid 1 to smithi20438004-1 2024-09-13T11:33:08.088 INFO:tasks.rados.rados.0.smithi204.stdout:1281: finishing write tid 2 to smithi20438004-1 2024-09-13T11:33:08.088 INFO:tasks.rados.rados.0.smithi204.stdout:1281: finishing write tid 3 to smithi20438004-1 2024-09-13T11:33:08.088 INFO:tasks.rados.rados.0.smithi204.stdout:1281: finishing write tid 4 to smithi20438004-1 2024-09-13T11:33:08.088 INFO:tasks.rados.rados.0.smithi204.stdout:1281: oid 1 updating version 0 to 66 2024-09-13T11:33:08.088 INFO:tasks.rados.rados.0.smithi204.stdout:1281: oid 1 version 66 is already newer than 65 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:1281: oid 1 version 66 is already newer than 64 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 66 (ObjNum 241 snap 0 seq_num 241) dirty exists 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:1281: left oid 1 (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:1283: finishing write tid 1 to smithi20438004-27 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:1280: done (12 left) 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:1281: done (11 left) 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:1282: done (10 left) 2024-09-13T11:33:08.089 INFO:tasks.rados.rados.0.smithi204.stdout:1293: set_chunk oid 8 target oid 2024-09-13T11:33:08.096 INFO:tasks.rados.rados.0.smithi204.stdout:1293: set_chunk oid 8 offset: 99840 length: 1705472 target oid a4d06837363597ff82ea6de2eedba3575a0b14b8eedbee6d4c464972a8dbc108 offset: 0 snap -1 2024-09-13T11:33:08.097 INFO:tasks.rados.rados.0.smithi204.stdout:1294: read oid 34 snap -1 2024-09-13T11:33:08.097 INFO:tasks.rados.rados.0.smithi204.stdout:1294: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:08.097 INFO:tasks.rados.rados.0.smithi204.stdout:1295: tier_promote oid 9 2024-09-13T11:33:08.097 INFO:tasks.rados.rados.0.smithi204.stdout:1285: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:08.112 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:08.111+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:08.117 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2042713 2024-09-13T11:33:08.139 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.140 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.149 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:08.213 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:08.211+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:08.234 INFO:tasks.rados.rados.0.smithi204.stdout:1284: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:33:08.234 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:08.294 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.294 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.303 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:08.314 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:08.312+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:08.328 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:08.376 INFO:tasks.rados.rados.0.smithi204.stdout:1283: finishing write tid 2 to smithi20438004-27 2024-09-13T11:33:08.376 INFO:tasks.rados.rados.0.smithi204.stdout:1283: finishing write tid 3 to smithi20438004-27 2024-09-13T11:33:08.376 INFO:tasks.rados.rados.0.smithi204.stdout:1283: finishing write tid 4 to smithi20438004-27 2024-09-13T11:33:08.376 INFO:tasks.rados.rados.0.smithi204.stdout:1283: oid 27 updating version 0 to 84 2024-09-13T11:33:08.376 INFO:tasks.rados.rados.0.smithi204.stdout:1283: oid 27 updating version 84 to 86 2024-09-13T11:33:08.377 INFO:tasks.rados.rados.0.smithi204.stdout:1283: oid 27 version 86 is already newer than 85 2024-09-13T11:33:08.377 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 86 (ObjNum 242 snap 0 seq_num 242) dirty exists 2024-09-13T11:33:08.377 INFO:tasks.rados.rados.0.smithi204.stdout:1283: left oid 27 (ObjNum 242 snap 0 seq_num 242) 2024-09-13T11:33:08.377 INFO:tasks.rados.rados.0.smithi204.stdout:1287: expect (ObjNum 197 snap 0 seq_num 197) 2024-09-13T11:33:08.377 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 28919 2024-09-13T11:33:08.408 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.409 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.415 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:08.414+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:08.417 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:08.447 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 417110 2024-09-13T11:33:08.517 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:08.515+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:08.552 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.552 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.560 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:08.586 INFO:tasks.rados.rados.0.smithi204.stdout:1286: expect (ObjNum 226 snap 0 seq_num 226) 2024-09-13T11:33:08.618 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:08.617+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:08.654 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:33:08.664 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.664 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.672 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:08.719 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:08.718+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:08.817 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.817 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.820 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:08.819+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:08.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:08.842 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:08.842 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:08.843 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:08.843 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T11:33:08.843 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-09-13T11:33:08.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1292: got (2) No such file or directory 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1292: got expected ENOENT (src dne) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 49 (ObjNum 129 snap 0 seq_num 3594327040) clean dne 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1283: done (12 left) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1284: done (11 left) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1285: done (10 left) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1286: done (9 left) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1287: done (8 left) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1290: done (7 left) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1292: done (6 left) 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1296: delete oid 5 current snap is 0 2024-09-13T11:33:08.846 INFO:tasks.rados.rados.0.smithi204.stdout:1288: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-13T11:33:08.921 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:08.920+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:08.947 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:08.947 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:08.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:08.967 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1167941 2024-09-13T11:33:09.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:09.012 INFO:tasks.rados.rados.0.smithi204.stdout:1289: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:09.012 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:09.021 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:09.020+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:09.080 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:09.080 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:09.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:09.134 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:09.132+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:09.137 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:09.137 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:09.155 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:09.156 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:09.156 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:09.173 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:09.173 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:09.189 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:09.189 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:09.201 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'23)", 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.418389972, 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005604375, 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:33:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779709+0000", 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.63e-07 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779721+0000", 2024-09-13T11:33:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.204e-05 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779728+0000", 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4150000000000002e-06 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779732+0000", 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6839999999999998e-06 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779761+0000", 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9405999999999999e-05 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779773+0000", 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1398e-05 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785264+0000", 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054913089999999998 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785313+0000", 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8659999999999998e-05 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 63.460918474000003, 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:33:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:33:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:33:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:33:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:33:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'105, mlcod=24'100)", 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 63.460354871, 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061709499999999997, 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737744+0000", 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7700000000000004e-07 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737749+0000", 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737756+0000", 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7649999999999996e-06 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737760+0000", 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6650000000000001e-06 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742613+0000", 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048528909999999998 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742629+0000", 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6481e-05 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743902+0000", 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012723630000000001 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743914+0000", 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2663999999999999e-05 2024-09-13T11:33:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.720992846999998, 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:33:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:33:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:33:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:33:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:33:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:33:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:33:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:33:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:33:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:33:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:33:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:33:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:33:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.715934289000003, 2024-09-13T11:33:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:33:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:33:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.715510467000001, 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:33:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:33:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:33:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:33:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.243 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.715492423000001, 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.244 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:33:09.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:33:09.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:33:09.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:33:09.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.999251651000002, 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.251 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:33:09.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.257 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.985410862000002, 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.258 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:33:09.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:33:09.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.886442887999998, 2024-09-13T11:33:09.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:33:09.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:33:09.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.b 51/50 [PushOp(4:da92e835:::smithi20438004-50:head, version: 38'19, data_included: [2616802~223548], data_size: 223548, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:da92e835:::smithi20438004-50:head@38'19, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2616802, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.885632941000001, 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055783109999999999, 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312466+0000", 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7499999999999999e-07 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312658+0000", 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019212100000000001 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312681+0000", 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3059999999999999e-05 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312694+0000", 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2408999999999999e-05 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312765+0000", 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1755000000000007e-05 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312775+0000", 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2879999999999998e-06 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.318043+0000", 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052689030000000001 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1751 4.9 4:9441befd:::smithi20438004-8:head [write 420208~530210,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.812530108000001, 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060468199999999996, 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.280 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.280 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.280 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.280 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1751 2024-09-13T11:33:09.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.280 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385569+0000", 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.426e-06 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386470+0000", 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090069800000000002 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386485+0000", 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5225000000000001e-05 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386495+0000", 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386574+0000", 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8911000000000004e-05 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386698+0000", 2024-09-13T11:33:09.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012372800000000001 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386913+0000", 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021553499999999999 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.388673+0000", 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017596210000000001 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.389582+0000", 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090943100000000004 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391260+0000", 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00167782 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391570+0000", 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030968499999999997 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391615+0000", 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5151999999999997e-05 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1752 4.9 4:9441befd:::smithi20438004-8:head [write 1527054~766050] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.804733950999999, 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062506259999999996, 2024-09-13T11:33:09.286 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1752 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:09.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393365+0000", 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0720000000000001e-06 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394108+0000", 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074250399999999995 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394120+0000", 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1990999999999999e-05 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394125+0000", 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7270000000000004e-06 2024-09-13T11:33:09.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394182+0000", 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6181000000000003e-05 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394248+0000", 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6848999999999996e-05 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394333+0000", 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4970999999999997e-05 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396066+0000", 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001732546 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397394+0000", 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013282680000000001 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399415+0000", 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002021063 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399585+0000", 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016999699999999999 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399615+0000", 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9456999999999999e-05 2024-09-13T11:33:09.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1753 4.9 4:9441befd:::smithi20438004-8:head [setxattr _header (55) in=7b,truncate 2420208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.803956864, 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055775579999999998, 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.293 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1753 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394142+0000", 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7999999999999995e-07 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394147+0000", 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2120000000000002e-06 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394154+0000", 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6980000000000001e-06 2024-09-13T11:33:09.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394160+0000", 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395493+0000", 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00133299 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395574+0000", 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0944000000000003e-05 2024-09-13T11:33:09.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395686+0000", 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112049 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396457+0000", 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077122900000000001 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397581+0000", 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011240989999999999 2024-09-13T11:33:09.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399655+0000", 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002074046 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399684+0000", 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8924999999999998e-05 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399719+0000", 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4459000000000003e-05 2024-09-13T11:33:09.298 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1754 4.9 4:9441befd:::smithi20438004-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.803926388000001, 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057025110000000004, 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.299 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1754 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2300000000000001e-07 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394175+0000", 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5189999999999999e-06 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394181+0000", 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6089999999999997e-06 2024-09-13T11:33:09.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394187+0000", 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7400000000000001e-06 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395928+0000", 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017413470000000001 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395983+0000", 2024-09-13T11:33:09.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399771+0000", 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003787786 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399814+0000", 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2755e-05 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399874+0000", 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0350999999999999e-05 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1959 4.e e53/51 4:7984cf84:::smithi20438004-46:head v 53'41, mlcod=52'38)", 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.009681618999998, 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057232109999999997, 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.304 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:09.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0599999999999999e-07 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188422+0000", 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9409999999999998e-06 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188434+0000", 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2002e-05 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188441+0000", 2024-09-13T11:33:09.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0090000000000001e-06 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188712+0000", 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027109600000000002 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188724+0000", 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1790999999999999e-05 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194092+0000", 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053680450000000001 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194140+0000", 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7521000000000001e-05 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.308 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2036 4.e e54/51 4:76a15ccf:::smithi20438004-1:head v 54'46, mlcod=54'44)", 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.620133114000001, 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056625620000000003, 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577966+0000", 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4200000000000005e-07 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578870+0000", 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090414100000000002 2024-09-13T11:33:09.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578880+0000", 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0501e-05 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578886+0000", 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.356e-06 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578950+0000", 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:33:09.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578968+0000", 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7935999999999999e-05 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583569+0000", 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046012680000000004 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583628+0000", 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8473000000000001e-05 2024-09-13T11:33:09.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2293 4.9 4:9441befd:::smithi20438004-8:head [write 1586353~512385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.926225174000001, 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066015969999999998, 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.313 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2293 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271874+0000", 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.497e-06 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272359+0000", 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048419700000000001 2024-09-13T11:33:09.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272399+0000", 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0754000000000003e-05 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272410+0000", 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272470+0000", 2024-09-13T11:33:09.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0603999999999998e-05 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272575+0000", 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104901 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272731+0000", 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015523300000000001 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.274392+0000", 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016616070000000001 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.276887+0000", 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024952360000000001 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278386+0000", 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014986540000000001 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278439+0000", 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2624999999999998e-05 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278475+0000", 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5812999999999997e-05 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2494 4.d e63/31 4:b15c1843:::smithi20438004-33:head v 63'65, mlcod=63'62)", 2024-09-13T11:33:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.5490114649999995, 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064248049999999996, 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649088+0000", 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.78e-07 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649096+0000", 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3790000000000006e-06 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649108+0000", 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1751e-05 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649116+0000", 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3850000000000002e-06 2024-09-13T11:33:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649307+0000", 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019067199999999999 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649325+0000", 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7859e-05 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655462+0000", 2024-09-13T11:33:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061369980000000003 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655511+0000", 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9883000000000003e-05 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.4.22:41 3.1 e63/51 3:b93f8350:::9968f6d703223c53b55081382c31f0a72d88ed4f88670426e6b8ab5df371e41b:head v 63'230, mlcod=63'227)", 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.0966419799999994, 2024-09-13T11:33:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063303989999999996, 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101457+0000", 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101463+0000", 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7529999999999998e-06 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101473+0000", 2024-09-13T11:33:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0154e-05 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101483+0000", 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101542+0000", 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9132999999999998e-05 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101560+0000", 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7156e-05 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107733+0000", 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061735740000000002 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107787+0000", 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3389e-05 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.329 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:09.330 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:09.330 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:09.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:09.331 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:09.332 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:09.332 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:09.332 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:09.332 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:09.333 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:09.333 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:09.333 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:09.334 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:09.334 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:09.334 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:09.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:09.335 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:09.335 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:09.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:09.336 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:09.328+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:09.343 INFO:tasks.rados.rados.0.smithi204.stdout:1293: finishing set_chunk to oid 8 2024-09-13T11:33:09.343 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 99840 length: 1705472 2024-09-13T11:33:09.343 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-13T11:33:09.343 INFO:tasks.rados.rados.0.smithi204.stdout:1291: expect (ObjNum 231 snap 0 seq_num 231) 2024-09-13T11:33:09.345 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:09.345 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:09.362 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:09.362 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:09.366 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:09.431 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:09.430+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:09.452 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:09.452 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:09.452 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:09.452 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:09.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:09.469 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1159448 2024-09-13T11:33:09.518 INFO:tasks.rados.rados.0.smithi204.stdout:1294: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:09.518 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:09.533 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:09.532+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:09.567 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:09.567 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:09.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:09.594 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:09.634 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:09.633+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:09.655 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T11:33:09.670 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:09.694 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:09.694 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:09.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:09.694 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:09.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.694 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.168470874, 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.695 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:33:09.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:33:09.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:26 4.8 4:1cf41451:::smithi20438004-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.162939461999997, 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083581760000000001, 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.705 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 26 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528196+0000", 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.048e-06 2024-09-13T11:33:09.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528601+0000", 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040467299999999998 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528612+0000", 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.171e-05 2024-09-13T11:33:09.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528620+0000", 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1140000000000006e-06 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532030+0000", 2024-09-13T11:33:09.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034098779999999999 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532128+0000", 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7910000000000003e-05 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532280+0000", 2024-09-13T11:33:09.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015168700000000001 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533907+0000", 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016271 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533914+0000", 2024-09-13T11:33:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2590000000000003e-06 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536475+0000", 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025609259999999998 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536517+0000", 2024-09-13T11:33:09.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.163e-05 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536553+0000", 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6241000000000001e-05 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:27 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.162502330000002, 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082102310000000001, 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.714 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 27 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:09.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:09.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.51e-07 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528635+0000", 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8399999999999999e-06 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528639+0000", 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528642+0000", 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.19e-06 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533286+0000", 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046437859999999996 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533371+0000", 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4900000000000004e-05 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533475+0000", 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104285 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534168+0000", 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069284800000000003 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534400+0000", 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023182699999999999 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536780+0000", 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023795880000000002 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536809+0000", 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9156e-05 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536842+0000", 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3546e-05 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.722 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:28 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.162477361000001, 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083368629999999999, 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.723 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 28 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528659+0000", 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528664+0000", 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.42e-06 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528675+0000", 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1447999999999999e-05 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533704+0000", 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050284769999999999 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533777+0000", 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3675000000000002e-05 2024-09-13T11:33:09.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536883+0000", 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031062820000000001 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536929+0000", 2024-09-13T11:33:09.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5237000000000001e-05 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536994+0000", 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5152000000000002e-05 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.731 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.731 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:09.731 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.479532732999999, 2024-09-13T11:33:09.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:33:09.731 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.731 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.732 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.732 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.732 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.732 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.732 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:33:09.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:09.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:33:09.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:33:09.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:33:09.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.741 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.741 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:09.741 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.471190534999998, 2024-09-13T11:33:09.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:33:09.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:09.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:33:09.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:33:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:33:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:33:09.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:33:09.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:33:09.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:33:09.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:33:09.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:33:09.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.450832532, 2024-09-13T11:33:09.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:33:09.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:33:09.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:33:09.757 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.443031773000001, 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.758 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.759 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:33:09.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:33:09.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:33:09.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:33:09.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:33:09.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:33:09.766 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.766 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.766 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.893524964999997, 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.767 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:33:09.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:33:09.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.774 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:797 4.8 4:1cf41451:::smithi20438004-7:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:09.775 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:09.775 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.880162402000003, 2024-09-13T11:33:09.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086937119999999993, 2024-09-13T11:33:09.775 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.776 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.776 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 797 2024-09-13T11:33:09.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:09.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810974+0000", 2024-09-13T11:33:09.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7010000000000001e-06 2024-09-13T11:33:09.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811111+0000", 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013712699999999999 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811123+0000", 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1854e-05 2024-09-13T11:33:09.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811131+0000", 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3979999999999999e-06 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813131+0000", 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019997740000000002 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813239+0000", 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108378 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813421+0000", 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018227099999999999 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814991+0000", 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015690599999999999 2024-09-13T11:33:09.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816465+0000", 2024-09-13T11:33:09.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001474784 2024-09-13T11:33:09.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819575+0000", 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031099769999999999 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819626+0000", 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0609999999999998e-05 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819666+0000", 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9777999999999998e-05 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:798 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:09.784 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.879988775999998, 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087603679999999993, 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 798 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.785 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811150+0000", 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811157+0000", 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811163+0000", 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813998+0000", 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002835382 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814069+0000", 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1099000000000005e-05 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814212+0000", 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142647 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816475+0000", 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022629799999999999 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816520+0000", 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5099999999999998e-05 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819739+0000", 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032186570000000002 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819820+0000", 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1527999999999996e-05 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819906+0000", 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5845000000000002e-05 2024-09-13T11:33:09.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:799 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.879947436000002, 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089993910000000007, 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.792 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.793 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.793 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.793 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 799 2024-09-13T11:33:09.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.793 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:09.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:09.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811188+0000", 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811192+0000", 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.566e-06 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811202+0000", 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4790000000000002e-06 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811209+0000", 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814715+0000", 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035061150000000002 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814798+0000", 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2854000000000003e-05 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819962+0000", 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051643419999999997 2024-09-13T11:33:09.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820067+0000", 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010525 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820186+0000", 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118886 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:805 4.8 4:1d367c64:::smithi20438004-45:head [read 0~2739508,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.879870095999998, 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082479100000000007, 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 805 2024-09-13T11:33:09.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811265+0000", 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5400000000000002e-07 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811266+0000", 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.939e-06 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811270+0000", 2024-09-13T11:33:09.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4319999999999999e-06 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811274+0000", 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4150000000000001e-06 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818160+0000", 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068854700000000003 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818409+0000", 2024-09-13T11:33:09.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024941299999999998 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819512+0000", 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011029869999999999 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:09.804 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.931336430000002, 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:09.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:33:09.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.811 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1485 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~3966419,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.580869333999999, 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01527358, 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1485 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.812 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:09.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:09.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2300000000000002e-07 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110269+0000", 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:33:09.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110274+0000", 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2349999999999999e-06 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110278+0000", 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6100000000000002e-06 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117718+0000", 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074403120000000001 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117821+0000", 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103501 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.125539+0000", 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077171590000000003 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.817 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1636 4.8 4:1d367c64:::smithi20438004-45:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.690702167000001, 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091050230000000003, 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.818 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1636 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000433+0000", 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1200000000000003e-07 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000436+0000", 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000443+0000", 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8619999999999997e-06 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000447+0000", 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8800000000000001e-06 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.001976+0000", 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001529774 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.002101+0000", 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124824 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009537+0000", 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074360850000000003 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1637 4.8 4:1d367c64:::smithi20438004-45:head [sparse-read 0~2368645] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.690679637999999, 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010163363, 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1637 2024-09-13T11:33:09.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000457+0000", 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000460+0000", 2024-09-13T11:33:09.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3720000000000001e-06 2024-09-13T11:33:09.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000463+0000", 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.225e-06 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009568+0000", 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091047630000000001 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009626+0000", 2024-09-13T11:33:09.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8142e-05 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.010618+0000", 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099181000000000009 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1749 4.8 4:1d367c64:::smithi20438004-45:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:09.830 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.903433472, 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49694346099999998, 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1749 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:09.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787703+0000", 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6840000000000001e-06 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787708+0000", 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787721+0000", 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2405000000000001e-05 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787728+0000", 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.391e-06 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787780+0000", 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1897000000000001e-05 2024-09-13T11:33:09.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787845+0000", 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5184000000000005e-05 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284644+0000", 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49679929099999998 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.835 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2310 4.18 e60/59 4:1cf41451:::smithi20438004-7:head v 60'49, mlcod=0'0)", 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.406437800000001, 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075680649999999997, 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284698+0000", 2024-09-13T11:33:09.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.043e-06 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285649+0000", 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095142200000000005 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285665+0000", 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6317999999999999e-05 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285674+0000", 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1969999999999996e-06 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285731+0000", 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7611000000000001e-05 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285747+0000", 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6211000000000001e-05 2024-09-13T11:33:09.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.292214+0000", 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064668670000000003 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.292265+0000", 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0396000000000003e-05 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2525 4.14 4:2c957d44:::smithi20438004-22:head [write 1782380~696402] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.092343+0000", 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.5987910240000005, 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084113729999999998, 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:09.841 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2525 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.092343+0000", 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:09.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.092343+0000", 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.092346+0000", 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0139999999999999e-06 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093044+0000", 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069733399999999997 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093057+0000", 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3134e-05 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093065+0000", 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1990000000000006e-06 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093131+0000", 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6416999999999998e-05 2024-09-13T11:33:09.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093238+0000", 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000106109 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093370+0000", 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013242899999999999 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.096417+0000", 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003046815 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.098025+0000", 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016082609999999999 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:09.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.100477+0000", 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002452105 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.100692+0000", 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000214861 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.100755+0000", 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2694999999999995e-05 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:09.848 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:09.848 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:09.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:09.851 INFO:tasks.rados.rados.0.smithi204.stdout:1295: got (0) Success 2024-09-13T11:33:09.851 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 32 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-13T11:33:09.851 INFO:tasks.rados.rados.0.smithi204.stdout:1288: done (6 left) 2024-09-13T11:33:09.851 INFO:tasks.rados.rados.0.smithi204.stdout:1289: done (5 left) 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1291: done (4 left) 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1293: done (3 left) 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1294: done (2 left) 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1295: done (1 left) 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1296: done (0 left) 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1297: read oid 18 snap -1 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1297: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1298: write (excl) oid 26 current snap is 0 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1298: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1298: writing smithi20438004-26 from 500908 to 984960 tid 1 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1298: writing smithi20438004-26 from 1411724 to 2100908 tid 2 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1299: set_chunk oid 49 target oid 2024-09-13T11:33:09.852 INFO:tasks.rados.rados.0.smithi204.stdout:1299: set_chunk oid 49 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:09.853 INFO:tasks.rados.rados.0.smithi204.stdout:1300: tier_promote oid 42 2024-09-13T11:33:09.853 INFO:tasks.rados.rados.0.smithi204.stdout:1297: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:33:09.853 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:09.847+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:09.899 INFO:tasks.rados.rados.0.smithi204.stdout:1298: finishing write tid 1 to smithi20438004-26 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1297: done (3 left) 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1301: read oid 23 snap -1 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1301: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1302: tier_flush oid 15 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1302: tier_flush oid 15 snap -1 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1299: finishing set_chunk to oid 49 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1299: got expected ENOENT (src dne) 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:1300: got (0) Success 2024-09-13T11:33:09.900 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 56 (ObjNum 186 snap 0 seq_num 186) dirty exists 2024-09-13T11:33:09.901 INFO:tasks.rados.rados.0.smithi204.stdout:1298: finishing write tid 2 to smithi20438004-26 2024-09-13T11:33:09.901 INFO:tasks.rados.rados.0.smithi204.stdout:1299: done (4 left) 2024-09-13T11:33:09.901 INFO:tasks.rados.rados.0.smithi204.stdout:1300: done (3 left) 2024-09-13T11:33:09.901 INFO:tasks.rados.rados.0.smithi204.stdout:1303: read oid 48 snap -1 2024-09-13T11:33:09.901 INFO:tasks.rados.rados.0.smithi204.stdout:1303: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-13T11:33:09.901 INFO:tasks.rados.rados.0.smithi204.stdout:1304: set_chunk oid 24 target oid 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1304: set_chunk oid 24 offset: 1001472 length: 291840 target oid e77f445921db11c02f1b781bb9ae5250947a4faf551aac69552c05efecdbf65c offset: 0 snap -1 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1305: read oid 13 snap -1 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1305: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1306: read oid 35 snap -1 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1306: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1307: write oid 12 current snap is 0 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1298: finishing write tid 3 to smithi20438004-26 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1298: finishing write tid 4 to smithi20438004-26 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1298: oid 26 updating version 0 to 52 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1298: oid 26 version 52 is already newer than 51 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:1298: oid 26 updating version 52 to 53 2024-09-13T11:33:09.905 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 53 (ObjNum 244 snap 0 seq_num 244) dirty exists 2024-09-13T11:33:09.906 INFO:tasks.rados.rados.0.smithi204.stdout:1298: left oid 26 (ObjNum 244 snap 0 seq_num 244) 2024-09-13T11:33:09.906 INFO:tasks.rados.rados.0.smithi204.stdout:1307: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-13T11:33:09.910 INFO:tasks.rados.rados.0.smithi204.stdout:1307: writing smithi20438004-12 from 517715 to 1077016 tid 1 2024-09-13T11:33:09.916 INFO:tasks.rados.rados.0.smithi204.stdout:1307: writing smithi20438004-12 from 1870206 to 2117715 tid 2 2024-09-13T11:33:09.916 INFO:tasks.rados.rados.0.smithi204.stdout:1298: done (7 left) 2024-09-13T11:33:09.916 INFO:tasks.rados.rados.0.smithi204.stdout:1308: write oid 45 current snap is 0 2024-09-13T11:33:09.916 INFO:tasks.rados.rados.0.smithi204.stdout:1302: got (2) No such file or directory 2024-09-13T11:33:09.917 INFO:tasks.rados.rados.0.smithi204.stdout:1302: got expected ENOENT (src dne) 2024-09-13T11:33:09.917 INFO:tasks.rados.rados.0.smithi204.stdout:1308: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-13T11:33:09.923 INFO:tasks.rados.rados.0.smithi204.stdout:1308: writing smithi20438004-45 from 534522 to 1169072 tid 1 2024-09-13T11:33:09.929 INFO:tasks.rados.rados.0.smithi204.stdout:1308: writing smithi20438004-45 from 1812335 to 2134522 tid 2 2024-09-13T11:33:09.930 INFO:tasks.rados.rados.0.smithi204.stdout:1302: done (7 left) 2024-09-13T11:33:09.930 INFO:tasks.rados.rados.0.smithi204.stdout:1309: set_chunk oid 10 target oid 2024-09-13T11:33:09.930 INFO:tasks.rados.rados.0.smithi204.stdout:1309: set_chunk oid 10 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:09.930 INFO:tasks.rados.rados.0.smithi204.stdout:1310: read oid 7 snap -1 2024-09-13T11:33:09.930 INFO:tasks.rados.rados.0.smithi204.stdout:1310: expect (ObjNum 197 snap 0 seq_num 197) 2024-09-13T11:33:09.930 INFO:tasks.rados.rados.0.smithi204.stdout:1311: tier_promote oid 38 2024-09-13T11:33:09.930 INFO:tasks.rados.rados.0.smithi204.stdout:1303: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-13T11:33:09.950 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:09.948+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:10.001 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:10.002 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:10.002 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:10.002 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:10.002 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2975276 2024-09-13T11:33:10.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:10.051 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:10.050+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:10.062 INFO:tasks.rados.rados.0.smithi204.stdout:1304: finishing set_chunk to oid 24 2024-09-13T11:33:10.062 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 85 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-09-13T11:33:10.062 INFO:tasks.rados.rados.0.smithi204.stdout:1306: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:10.117 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:10.117 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:10.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:10.137 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2275628 2024-09-13T11:33:10.152 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:10.151+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:10.229 INFO:tasks.rados.rados.0.smithi204.stdout:1301: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-13T11:33:10.229 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:10.238 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.829723375, 2024-09-13T11:33:10.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026017369999999998, 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402441+0000", 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.508e-06 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403463+0000", 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010217970000000001 2024-09-13T11:33:10.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403475+0000", 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2498e-05 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403483+0000", 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4590000000000001e-06 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403531+0000", 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-05 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403552+0000", 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0270000000000001e-05 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.404996+0000", 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001443915 2024-09-13T11:33:10.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.405041+0000", 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5540000000000001e-05 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.823958384000001, 2024-09-13T11:33:10.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028959070000000001, 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408205+0000", 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409400+0000", 2024-09-13T11:33:10.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011945110000000001 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409410+0000", 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0159999999999999e-05 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409416+0000", 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409467+0000", 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1270999999999998e-05 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409489+0000", 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1682999999999999e-05 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411056+0000", 2024-09-13T11:33:10.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015677569999999999 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411100+0000", 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4036999999999999e-05 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.250 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1728 4.0 e50/49 4:04aeeab9:::smithi20438004-5:head v 50'45, mlcod=0'0)", 2024-09-13T11:33:10.250 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:33:10.250 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.787903042, 2024-09-13T11:33:10.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002646874, 2024-09-13T11:33:10.250 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.251 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:33:10.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:33:10.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444261+0000", 2024-09-13T11:33:10.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0920000000000001e-06 2024-09-13T11:33:10.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445429+0000", 2024-09-13T11:33:10.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00116847 2024-09-13T11:33:10.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445444+0000", 2024-09-13T11:33:10.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4487999999999999e-05 2024-09-13T11:33:10.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445454+0000", 2024-09-13T11:33:10.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8349999999999999e-06 2024-09-13T11:33:10.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445509+0000", 2024-09-13T11:33:10.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5154000000000001e-05 2024-09-13T11:33:10.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445525+0000", 2024-09-13T11:33:10.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6654e-05 2024-09-13T11:33:10.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.446871+0000", 2024-09-13T11:33:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001345534 2024-09-13T11:33:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.446907+0000", 2024-09-13T11:33:10.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5646999999999999e-05 2024-09-13T11:33:10.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.260 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.260 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1904 4.5 e53/50 4:a5f8f9e2:::smithi20438004-47:head v 53'37, mlcod=52'35)", 2024-09-13T11:33:10.261 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:10.261 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.299221244999998, 2024-09-13T11:33:10.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025939650000000002, 2024-09-13T11:33:10.261 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.261 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.262 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.262 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:10.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:10.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:10.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.66e-07 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933607+0000", 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00066545899999999997 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933617+0000", 2024-09-13T11:33:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0273e-05 2024-09-13T11:33:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933625+0000", 2024-09-13T11:33:10.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5079999999999999e-06 2024-09-13T11:33:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933679+0000", 2024-09-13T11:33:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4447999999999998e-05 2024-09-13T11:33:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933701+0000", 2024-09-13T11:33:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1342999999999999e-05 2024-09-13T11:33:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.935495+0000", 2024-09-13T11:33:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001794604 2024-09-13T11:33:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.935536+0000", 2024-09-13T11:33:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0163999999999997e-05 2024-09-13T11:33:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.257977132000001, 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026856010000000001, 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3300000000000002e-07 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974977+0000", 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079094699999999996 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974994+0000", 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7085e-05 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975004+0000", 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3149999999999998e-06 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975070+0000", 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6167999999999994e-05 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975097+0000", 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6855e-05 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976824+0000", 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017271039999999999 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976871+0000", 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7494000000000003e-05 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.313054218000001, 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027642500000000002, 2024-09-13T11:33:10.276 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3200000000000001e-07 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919918+0000", 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080951299999999999 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919931+0000", 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2662e-05 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919940+0000", 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3780000000000007e-06 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.920022+0000", 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2057000000000004e-05 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.920044+0000", 2024-09-13T11:33:10.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1863999999999999e-05 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.921822+0000", 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017779779999999999 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.921873+0000", 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0565999999999997e-05 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2077 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'67, mlcod=53'65)", 2024-09-13T11:33:10.282 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.927506+0000", 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.304656585, 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002456813, 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.927506+0000", 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.927506+0000", 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.927508+0000", 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.345e-06 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.928154+0000", 2024-09-13T11:33:10.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064647100000000002 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.928171+0000", 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7422999999999999e-05 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.928180+0000", 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5799999999999992e-06 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.928243+0000", 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2434999999999994e-05 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.928274+0000", 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1353000000000003e-05 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.929933+0000", 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016592200000000001 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.929963+0000", 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9986000000000001e-05 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2315 4.0 e58/56 4:001d90c2:::smithi20438004-28:head v 58'53, mlcod=57'51)", 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.112500575, 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031817899999999999, 2024-09-13T11:33:10.287 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119664+0000", 2024-09-13T11:33:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8330000000000001e-06 2024-09-13T11:33:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120926+0000", 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012617710000000001 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120940+0000", 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4544e-05 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120952+0000", 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1676e-05 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121050+0000", 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8177999999999994e-05 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121071+0000", 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1174000000000001e-05 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122794+0000", 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00172215 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122844+0000", 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0464000000000003e-05 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2353 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'51, mlcod=0'0)", 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.306096+0000", 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.926066413000001, 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027573580000000001, 2024-09-13T11:33:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.306096+0000", 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.306096+0000", 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.306097+0000", 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307358+0000", 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012603720000000001 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307394+0000", 2024-09-13T11:33:10.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6668999999999997e-05 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307403+0000", 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4139999999999994e-06 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307471+0000", 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8088000000000003e-05 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307497+0000", 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6077000000000001e-05 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.308807+0000", 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001310121 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.308854+0000", 2024-09-13T11:33:10.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6826000000000002e-05 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4b088bf8:::smithi20438004-24:head, version: 60'85, data_included: [540672~892928,1630208~155648], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4b088bf8:::smithi20438004-24:head@60'85, size: 3058874, copy_subset: [(0, 3058874)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1785856, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.918972993000001, 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003437201, 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:10.299 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313191+0000", 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.046e-06 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315519+0000", 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023280670000000001 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315551+0000", 2024-09-13T11:33:10.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2440000000000001e-05 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315574+0000", 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2286e-05 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315668+0000", 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929000000000001e-05 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315679+0000", 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1e-05 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.316627+0000", 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094843300000000005 2024-09-13T11:33:10.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2354 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'52, mlcod=57'50)", 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.910976843, 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058622930000000002, 2024-09-13T11:33:10.304 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321187+0000", 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2200000000000003e-07 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323519+0000", 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002331891 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323532+0000", 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3494e-05 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323542+0000", 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323612+0000", 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9809999999999997e-05 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323638+0000", 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6526999999999999e-05 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.326990+0000", 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033519270000000002 2024-09-13T11:33:10.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327048+0000", 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8158999999999997e-05 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2355 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'53, mlcod=57'50)", 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:10.309 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.907928468, 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028910490000000001, 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324235+0000", 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6699999999999999e-07 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324282+0000", 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7437e-05 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324290+0000", 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3170000000000002e-06 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324297+0000", 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.691e-06 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324944+0000", 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064681799999999996 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324977+0000", 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2591e-05 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327083+0000", 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021065540000000001 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327125+0000", 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2274000000000002e-05 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2356 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'54, mlcod=57'50)", 2024-09-13T11:33:10.314 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.907854802999999, 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030409059999999999, 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0100000000000001e-07 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324312+0000", 2024-09-13T11:33:10.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324318+0000", 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6119999999999998e-06 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324323+0000", 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325363+0000", 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001040116 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325383+0000", 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9644999999999999e-05 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327310+0000", 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001927205 2024-09-13T11:33:10.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327349+0000", 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8930000000000002e-05 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2513 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'86, mlcod=0'0)", 2024-09-13T11:33:10.319 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.1663328150000005, 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034473500000000001, 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8099999999999999e-07 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066800+0000", 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096952400000000004 2024-09-13T11:33:10.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066816+0000", 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6010000000000001e-05 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066826+0000", 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0679000000000001e-05 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066970+0000", 2024-09-13T11:33:10.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014400900000000001 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.067020+0000", 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9595000000000002e-05 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069220+0000", 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022001640000000001 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069277+0000", 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7188000000000002e-05 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2515 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'88, mlcod=63'86)", 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.1531182080000004, 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007260702, 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.324 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4300000000000002e-07 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080302+0000", 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001256889 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080314+0000", 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.134e-05 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080319+0000", 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6699999999999999e-06 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080371+0000", 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2039000000000001e-05 2024-09-13T11:33:10.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080397+0000", 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5996e-05 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086251+0000", 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058537789999999999 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086305+0000", 2024-09-13T11:33:10.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4146e-05 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2516 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'89, mlcod=63'86)", 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.1517420279999993, 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062074870000000002, 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.329 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:10.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7199999999999999e-07 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080426+0000", 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080434+0000", 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2930000000000001e-06 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080443+0000", 2024-09-13T11:33:10.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3699999999999995e-06 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081319+0000", 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087608100000000002 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081346+0000", 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7282999999999999e-05 2024-09-13T11:33:10.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086595+0000", 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052492759999999998 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086628+0000", 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2836999999999997e-05 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.333 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.14.0:4 4.12 e63/61 4:4b088bf8:::smithi20438004-24:head v 63'90, mlcod=63'88)", 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.123612+0000", 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.1085506330000001, 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025216280000000002, 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.123612+0000", 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.123612+0000", 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.123613+0000", 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1519999999999999e-06 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.124982+0000", 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001368527 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.124997+0000", 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5407999999999999e-05 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.125005+0000", 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4100000000000002e-06 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.125061+0000", 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6209999999999999e-05 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.125092+0000", 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0948999999999998e-05 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.126104+0000", 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010122659999999999 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.126134+0000", 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9706e-05 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2718 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'58, mlcod=58'56)", 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1083432750000002, 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039829779999999999, 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.339 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:10.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125730+0000", 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001910213 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125744+0000", 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3326000000000001e-05 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125752+0000", 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125811+0000", 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9582e-05 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125836+0000", 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4482e-05 2024-09-13T11:33:10.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127747+0000", 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019112599999999999 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127803+0000", 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5633000000000001e-05 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2719 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'59, mlcod=60'57)", 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1005660150000001, 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032139159999999998, 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131598+0000", 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7999999999999993e-07 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132663+0000", 2024-09-13T11:33:10.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010650010000000001 2024-09-13T11:33:10.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132690+0000", 2024-09-13T11:33:10.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6970000000000001e-05 2024-09-13T11:33:10.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132703+0000", 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.36e-05 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132837+0000", 2024-09-13T11:33:10.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013366699999999999 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132900+0000", 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3331999999999996e-05 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134769+0000", 2024-09-13T11:33:10.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018688520000000001 2024-09-13T11:33:10.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134811+0000", 2024-09-13T11:33:10.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1514000000000003e-05 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2720 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'60, mlcod=66'58)", 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.138205+0000", 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0939578440000002, 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025807759999999999, 2024-09-13T11:33:10.351 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:10.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:10.352 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.352 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:10.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:10.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.138205+0000", 2024-09-13T11:33:10.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.138205+0000", 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.138206+0000", 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1599999999999999e-06 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139281+0000", 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010749760000000001 2024-09-13T11:33:10.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139294+0000", 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2807e-05 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139300+0000", 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2940000000000002e-06 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139349+0000", 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8965999999999998e-05 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139363+0000", 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3791999999999999e-05 2024-09-13T11:33:10.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.140744+0000", 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013805110000000001 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.140786+0000", 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.227e-05 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:10.357 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:10.358 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:10.359 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:10.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:10.360 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:10.358+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:10.448 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:10.460 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:10.459+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:10.474 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:10.474 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:10.474 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:10.474 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:10.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:10.557 INFO:tasks.rados.rados.0.smithi204.stdout:1305: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:10.561 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:10.560+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:10.626 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:10.626 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:10.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:10.635 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:10.662 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:10.659+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:10.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:33:10.739 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:10.739 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:10.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:10.739 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:10.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.739 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.645714523999999, 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011006991000000001, 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:10.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091279+0000", 2024-09-13T11:33:10.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.091e-06 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098180+0000", 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069015359999999998 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098194+0000", 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.323e-05 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098202+0000", 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2879999999999993e-06 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098267+0000", 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4733999999999994e-05 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098296+0000", 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:33:10.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102225+0000", 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039281940000000003 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102285+0000", 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.588909035, 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010863593, 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148085+0000", 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2810000000000001e-06 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153254+0000", 2024-09-13T11:33:10.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051698430000000004 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153266+0000", 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1973e-05 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153274+0000", 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153331+0000", 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6492000000000001e-05 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153396+0000", 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5083e-05 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158906+0000", 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055099570000000002 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158947+0000", 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.749 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.583696488999998, 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096623879999999992, 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157238+0000", 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039414139999999999 2024-09-13T11:33:10.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157253+0000", 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5566e-05 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157260+0000", 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.477e-06 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157315+0000", 2024-09-13T11:33:10.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4857e-05 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157341+0000", 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162899+0000", 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055583480000000003 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162958+0000", 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9169000000000002e-05 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.754 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.574038404, 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011410943999999999, 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162955+0000", 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0669999999999999e-06 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167312+0000", 2024-09-13T11:33:10.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004357072 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167333+0000", 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0842000000000001e-05 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167341+0000", 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4430000000000003e-06 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167406+0000", 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4245000000000006e-05 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167433+0000", 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7500000000000001e-05 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174280+0000", 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068473969999999999 2024-09-13T11:33:10.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174365+0000", 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4377999999999997e-05 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.561210641000002, 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099135219999999993, 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.759 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175783+0000", 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179929+0000", 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041461989999999997 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179943+0000", 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4731e-05 2024-09-13T11:33:10.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179953+0000", 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180049+0000", 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6909000000000003e-05 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180073+0000", 2024-09-13T11:33:10.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4077e-05 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185641+0000", 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055674330000000001 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185695+0000", 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4273e-05 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:258 4.3 4:c717f8db:::smithi20438004-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.523443374999999, 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017162947000000001, 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 258 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.764 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213551+0000", 2024-09-13T11:33:10.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0760000000000001e-06 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213566+0000", 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5441999999999999e-05 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213585+0000", 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8623000000000001e-05 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213600+0000", 2024-09-13T11:33:10.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4783999999999999e-05 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213667+0000", 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213747+0000", 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9794000000000002e-05 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:10.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228488+0000", 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014740964 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229259+0000", 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077144599999999996 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229743+0000", 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048437000000000003 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230652+0000", 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090904600000000005 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230688+0000", 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5474e-05 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230712+0000", 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3954000000000001e-05 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:259 4.1 4:8533c921:::smithi20438004-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.519167772999999, 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014548484, 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 259 2024-09-13T11:33:10.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217826+0000", 2024-09-13T11:33:10.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6360000000000001e-06 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217833+0000", 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8700000000000003e-06 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217844+0000", 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217851+0000", 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217896+0000", 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4808000000000002e-05 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217954+0000", 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7856000000000003e-05 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229813+0000", 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011859533 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231353+0000", 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001539503 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232188+0000", 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083500600000000001 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232284+0000", 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6645999999999996e-05 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232325+0000", 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0071999999999999e-05 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232373+0000", 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8498999999999998e-05 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:266 4.4 4:2451796c:::smithi20438004-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.514667977000002, 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099464859999999992, 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.776 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 266 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222325+0000", 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222330+0000", 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999997e-06 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222341+0000", 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0528e-05 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222347+0000", 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6409999999999996e-06 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222406+0000", 2024-09-13T11:33:10.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8449000000000002e-05 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222487+0000", 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1167000000000004e-05 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230197+0000", 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077105120000000001 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230937+0000", 2024-09-13T11:33:10.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000740159 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231296+0000", 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035862399999999999 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232186+0000", 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089002899999999997 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232234+0000", 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7414999999999999e-05 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232271+0000", 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7323000000000002e-05 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:283 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.782 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.503539842000002, 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011861159, 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 283 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233453+0000", 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233457+0000", 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1150000000000004e-06 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233465+0000", 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233471+0000", 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3450000000000002e-06 2024-09-13T11:33:10.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233526+0000", 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4790000000000002e-05 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233602+0000", 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6253000000000003e-05 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237634+0000", 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040318580000000001 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238163+0000", 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052849899999999998 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242711+0000", 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045478530000000001 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245235+0000", 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025239160000000002 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245286+0000", 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1739e-05 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245314+0000", 2024-09-13T11:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7280999999999999e-05 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:284 4.1 4:8b052c81:::smithi20438004-35:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.503189829999997, 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011653294, 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.789 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 284 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233803+0000", 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4600000000000003e-07 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233808+0000", 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:33:10.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233817+0000", 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9870000000000005e-06 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233824+0000", 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8290000000000002e-06 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233874+0000", 2024-09-13T11:33:10.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9877999999999999e-05 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233943+0000", 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238024+0000", 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040810350000000002 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238440+0000", 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041507799999999999 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242785+0000", 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043457890000000001 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245363+0000", 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025773359999999999 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245416+0000", 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3156999999999999e-05 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245456+0000", 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0000000000000003e-05 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.795 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:286 4.5 4:a9b0c9bd:::smithi20438004-37:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.502208068999998, 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013081485, 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.796 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 286 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234785+0000", 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234790+0000", 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.651e-06 2024-09-13T11:33:10.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234800+0000", 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234807+0000", 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.861e-06 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234868+0000", 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0582999999999999e-05 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234967+0000", 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054743730000000003 2024-09-13T11:33:10.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241205+0000", 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076331700000000001 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241351+0000", 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619399999999999 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247766+0000", 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064151290000000003 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247825+0000", 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8739999999999997e-05 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247866+0000", 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747000000000001e-05 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:290 4.1 4:8d6fdfdb:::smithi20438004-41:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.497653919999998, 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014414148999999999, 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 290 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239339+0000", 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:33:10.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239344+0000", 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2290000000000002e-06 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239354+0000", 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239361+0000", 2024-09-13T11:33:10.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.092e-06 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239406+0000", 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126000000000001e-05 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239452+0000", 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6192000000000001e-05 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248478+0000", 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090262209999999992 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249309+0000", 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083057300000000003 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253400+0000", 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040908760000000002 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253662+0000", 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026229500000000001 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253715+0000", 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2836e-05 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253753+0000", 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7526e-05 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.808 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:292 4.1 4:84fa1f64:::smithi20438004-43:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.497230707, 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010951834000000001, 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 292 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.809 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:10.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8800000000000002e-07 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239765+0000", 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239773+0000", 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8229999999999995e-06 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239779+0000", 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9900000000000002e-06 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239831+0000", 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2326e-05 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239910+0000", 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8623999999999998e-05 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245574+0000", 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005663927 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.246099+0000", 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052478300000000002 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247963+0000", 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001864166 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250625+0000", 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002662228 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250676+0000", 2024-09-13T11:33:10.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0989000000000003e-05 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250713+0000", 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7360000000000001e-05 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:317 4.1 4:8c30e60e:::smithi20438004-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:10.815 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.486106802000002, 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013134860999999999, 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 317 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250892+0000", 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250900+0000", 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250908+0000", 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.337e-06 2024-09-13T11:33:10.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250967+0000", 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8201000000000001e-05 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251045+0000", 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299000000000004e-05 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258769+0000", 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007724323 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259599+0000", 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082958200000000004 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260992+0000", 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00139291 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263918+0000", 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029263190000000001 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263971+0000", 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2670999999999998e-05 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264020+0000", 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.821 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:330 4.3 4:c7f26bfe:::smithi20438004-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.484883987000003, 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011943621, 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.822 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 330 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252109+0000", 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252116+0000", 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9199999999999998e-06 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252129+0000", 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2768e-05 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252138+0000", 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5939999999999994e-06 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252205+0000", 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252280+0000", 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4930000000000003e-05 2024-09-13T11:33:10.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259484+0000", 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00720434 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259994+0000", 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051054200000000001 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260333+0000", 2024-09-13T11:33:10.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033855000000000002 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263947+0000", 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00361452 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264015+0000", 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7155000000000003e-05 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264052+0000", 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305000000000003e-05 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:333 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.484593791000002, 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011814712, 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.828 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 333 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252406+0000", 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252416+0000", 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-05 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252423+0000", 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252486+0000", 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2805999999999995e-05 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252540+0000", 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3897000000000003e-05 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259138+0000", 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065977429999999997 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259785+0000", 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064727700000000003 2024-09-13T11:33:10.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264056+0000", 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042706710000000002 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264152+0000", 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5877000000000004e-05 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264180+0000", 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8674999999999999e-05 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264213+0000", 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3015000000000002e-05 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:433 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.460042944000001, 2024-09-13T11:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098551839999999995, 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276950+0000", 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999995e-07 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276955+0000", 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1059999999999999e-06 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276965+0000", 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276977+0000", 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1518000000000001e-05 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277035+0000", 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8133000000000001e-05 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277125+0000", 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0148999999999994e-05 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282235+0000", 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005109461 2024-09-13T11:33:10.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282725+0000", 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049045700000000002 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286634+0000", 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039087929999999998 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286707+0000", 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2692000000000001e-05 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286772+0000", 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5828999999999996e-05 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286805+0000", 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2134000000000001e-05 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.840 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:791 4.5 4:a5f8f9e2:::smithi20438004-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.037160034000003, 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019917450999999999, 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:10.841 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 791 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699834+0000", 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699840+0000", 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5989999999999998e-06 2024-09-13T11:33:10.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699852+0000", 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699861+0000", 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5259999999999993e-06 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699949+0000", 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8083000000000004e-05 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699953+0000", 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3869999999999997e-06 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701189+0000", 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012354740000000001 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701250+0000", 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1733e-05 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.719750+0000", 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018499504 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1374 4.1 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.657092906000003, 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011338608999999999, 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1374 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079900+0000", 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-07 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079901+0000", 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-13T11:33:10.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079905+0000", 2024-09-13T11:33:10.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2039999999999999e-06 2024-09-13T11:33:10.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079908+0000", 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8200000000000001e-06 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081697+0000", 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017893080000000001 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081802+0000", 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105231 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.091238+0000", 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094355819999999997 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1454 4.1 4:84fa1f64:::smithi20438004-43:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.901046063000003, 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025598754000000001, 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1454 2024-09-13T11:33:10.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835947+0000", 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8500000000000002e-07 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835948+0000", 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:10.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835952+0000", 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4259999999999999e-06 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835954+0000", 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8550000000000002e-06 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836019+0000", 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4416000000000001e-05 2024-09-13T11:33:10.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836271+0000", 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025171299999999998 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.861545+0000", 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025274496 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:10.859 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:10.859 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:10.860 INFO:tasks.rados.rados.0.smithi204.stdout:1307: finishing write tid 1 to smithi20438004-12 2024-09-13T11:33:10.860 INFO:tasks.rados.rados.0.smithi204.stdout:1307: finishing write tid 2 to smithi20438004-12 2024-09-13T11:33:10.860 INFO:tasks.rados.rados.0.smithi204.stdout:1307: finishing write tid 3 to smithi20438004-12 2024-09-13T11:33:10.860 INFO:tasks.rados.rados.0.smithi204.stdout:1307: finishing write tid 4 to smithi20438004-12 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1307: oid 12 updating version 0 to 74 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1307: oid 12 updating version 74 to 75 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1307: oid 12 version 75 is already newer than 73 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 75 (ObjNum 245 snap 0 seq_num 245) dirty exists 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1307: left oid 12 (ObjNum 245 snap 0 seq_num 245) 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1309: finishing set_chunk to oid 10 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1309: got expected ENOENT (src dne) 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1311: got (0) Success 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 102 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:1310: expect (ObjNum 197 snap 0 seq_num 197) 2024-09-13T11:33:10.861 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:10.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:10.863 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:10.859+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:10.961 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:10.959+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:10.966 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:10.966 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:10.966 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:10.966 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:10.970 INFO:tasks.rados.rados.0.smithi204.stdout:1308: finishing write tid 1 to smithi20438004-45 2024-09-13T11:33:10.970 INFO:tasks.rados.rados.0.smithi204.stdout:1308: finishing write tid 2 to smithi20438004-45 2024-09-13T11:33:10.970 INFO:tasks.rados.rados.0.smithi204.stdout:1308: finishing write tid 3 to smithi20438004-45 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1308: finishing write tid 4 to smithi20438004-45 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1308: oid 45 updating version 0 to 66 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1308: oid 45 version 66 is already newer than 65 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1308: oid 45 updating version 66 to 67 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 67 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1308: left oid 45 (ObjNum 246 snap 0 seq_num 246) 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1301: done (9 left) 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1303: done (8 left) 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1304: done (7 left) 2024-09-13T11:33:10.971 INFO:tasks.rados.rados.0.smithi204.stdout:1305: done (6 left) 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1306: done (5 left) 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1307: done (4 left) 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1308: done (3 left) 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1309: done (2 left) 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1310: done (1 left) 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1311: done (0 left) 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1312: write oid 39 current snap is 0 2024-09-13T11:33:10.972 INFO:tasks.rados.rados.0.smithi204.stdout:1312: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-13T11:33:10.975 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:10.976 INFO:tasks.rados.rados.0.smithi204.stdout:1312: writing smithi20438004-39 from 551329 to 1261128 tid 1 2024-09-13T11:33:10.980 INFO:tasks.rados.rados.0.smithi204.stdout:1312: writing smithi20438004-39 from 1870817 to 2151329 tid 2 2024-09-13T11:33:10.980 INFO:tasks.rados.rados.0.smithi204.stdout:1313: tier_promote oid 3 2024-09-13T11:33:10.980 INFO:tasks.rados.rados.0.smithi204.stdout:1314: write oid 26 current snap is 0 2024-09-13T11:33:10.980 INFO:tasks.rados.rados.0.smithi204.stdout:1314: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-13T11:33:10.987 INFO:tasks.rados.rados.0.smithi204.stdout:1314: writing smithi20438004-26 from 568136 to 1353184 tid 1 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:1314: writing smithi20438004-26 from 1929299 to 2168136 tid 2 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:1315: tier_promote oid 14 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:1312: finishing write tid 1 to smithi20438004-39 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:1313: got (2) No such file or directory 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:1313: got expected ENOENT (src dne) 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 28 (ObjNum 129 snap 0 seq_num 3594326576) clean dne 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:1312: finishing write tid 2 to smithi20438004-39 2024-09-13T11:33:10.993 INFO:tasks.rados.rados.0.smithi204.stdout:1312: finishing write tid 3 to smithi20438004-39 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1312: finishing write tid 4 to smithi20438004-39 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1312: oid 39 updating version 0 to 87 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1312: oid 39 updating version 87 to 89 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1312: oid 39 version 89 is already newer than 88 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 89 (ObjNum 247 snap 0 seq_num 247) dirty exists 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1312: left oid 39 (ObjNum 247 snap 0 seq_num 247) 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1312: done (3 left) 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1313: done (2 left) 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1316: set_chunk oid 37 target oid 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1316: set_chunk oid 37 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:10.994 INFO:tasks.rados.rados.0.smithi204.stdout:1317: set_chunk oid 30 target oid 2024-09-13T11:33:11.001 INFO:tasks.rados.rados.0.smithi204.stdout:1317: set_chunk oid 30 offset: 967680 length: 332288 target oid 70f160487d43a007968fc84c21e5a60e6d7d1103d74fde826faa63c1dded1d4d offset: 0 snap -1 2024-09-13T11:33:11.001 INFO:tasks.rados.rados.0.smithi204.stdout:1318: read oid 33 snap -1 2024-09-13T11:33:11.001 INFO:tasks.rados.rados.0.smithi204.stdout:1318: expect (ObjNum 215 snap 0 seq_num 215) 2024-09-13T11:33:11.001 INFO:tasks.rados.rados.0.smithi204.stdout:1319: read oid 6 snap -1 2024-09-13T11:33:11.001 INFO:tasks.rados.rados.0.smithi204.stdout:1319: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1320: tier_evict oid 36 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1320: tier_evict oid 36 snap -1 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1316: finishing set_chunk to oid 37 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1316: got expected ENOENT (src dne) 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1316: done (6 left) 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1321: read oid 2 snap -1 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1321: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1322: read oid 7 snap -1 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1322: expect (ObjNum 197 snap 0 seq_num 197) 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1323: delete oid 41 current snap is 0 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1315: got (2) No such file or directory 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1315: got expected ENOENT (src dne) 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 89 (ObjNum 33 snap 0 seq_num 0) clean dne 2024-09-13T11:33:11.002 INFO:tasks.rados.rados.0.smithi204.stdout:1314: finishing write tid 1 to smithi20438004-26 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1314: finishing write tid 2 to smithi20438004-26 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1314: finishing write tid 3 to smithi20438004-26 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1314: finishing write tid 4 to smithi20438004-26 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1314: oid 26 updating version 0 to 56 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1314: oid 26 version 56 is already newer than 54 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1314: oid 26 version 56 is already newer than 55 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 56 (ObjNum 248 snap 0 seq_num 248) dirty exists 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1314: left oid 26 (ObjNum 248 snap 0 seq_num 248) 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1317: finishing set_chunk to oid 30 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 967680 length: 332288 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-13T11:33:11.003 INFO:tasks.rados.rados.0.smithi204.stdout:1320: got (22) Invalid argument 2024-09-13T11:33:11.004 INFO:tasks.rados.rados.0.smithi204.stdout:1322: expect (ObjNum 197 snap 0 seq_num 197) 2024-09-13T11:33:11.062 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:11.061+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:11.078 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:11.078 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:11.078 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:11.078 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:11.078 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:11.087 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:11.119 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2848005 2024-09-13T11:33:11.162 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:11.160+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:11.173 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2712530 2024-09-13T11:33:11.186 INFO:tasks.rados.rados.0.smithi204.stdout:1319: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-13T11:33:11.190 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:11.190 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.845678061000001, 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079674329999999995, 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4344", 2024-09-13T11:33:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343169+0000", 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343177+0000", 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4819999999999994e-06 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343189+0000", 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2306999999999999e-05 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343199+0000", 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343279+0000", 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0025999999999996e-05 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343289+0000", 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0518e-05 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348428+0000", 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051380779999999999 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348510+0000", 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2226999999999997e-05 2024-09-13T11:33:11.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348598+0000", 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8722999999999997e-05 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349014+0000", 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041537900000000001 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349279+0000", 2024-09-13T11:33:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026517700000000001 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351081+0000", 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001802292 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351110+0000", 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8651000000000001e-05 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351132+0000", 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069000000000002e-05 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:21 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'4, mlcod=38'2)", 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.675695009000002, 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002707297, 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.198 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513149+0000", 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.322e-06 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514452+0000", 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303153 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514468+0000", 2024-09-13T11:33:11.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6045999999999998e-05 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514478+0000", 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4960000000000002e-06 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514531+0000", 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3442000000000001e-05 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514546+0000", 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5211999999999999e-05 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515821+0000", 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001274854 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515855+0000", 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3772000000000001e-05 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.202 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.167468511000003, 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002340637, 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021376+0000", 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.013e-06 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022454+0000", 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010781110000000001 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022469+0000", 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4199000000000001e-05 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022477+0000", 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022532+0000", 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5636000000000001e-05 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022548+0000", 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5769000000000001e-05 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023686+0000", 2024-09-13T11:33:11.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011379929999999999 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023715+0000", 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8659000000000001e-05 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:182 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'8, mlcod=38'6)", 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.161402838000001, 2024-09-13T11:33:11.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022898689999999999, 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027441+0000", 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7599999999999996e-07 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028556+0000", 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011154439999999999 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028566+0000", 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8269999999999993e-06 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028571+0000", 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8550000000000001e-06 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028614+0000", 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028629+0000", 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029703+0000", 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001073898 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029730+0000", 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6993999999999999e-05 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.211 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:189 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'10, mlcod=38'8)", 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.138981534999999, 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020004459999999999, 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049861+0000", 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.049862+0000", 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2999999999999999e-07 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050772+0000", 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091040000000000001 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050787+0000", 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4021e-05 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050793+0000", 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6959999999999999e-06 2024-09-13T11:33:11.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050843+0000", 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0148999999999997e-05 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.050860+0000", 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6665999999999999e-05 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.051831+0000", 2024-09-13T11:33:11.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097091600000000005 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.051862+0000", 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0667999999999998e-05 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:190 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'11, mlcod=38'9)", 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.131575109000003, 2024-09-13T11:33:11.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022963139999999998, 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058460+0000", 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192238 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058470+0000", 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058475+0000", 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5399999999999997e-06 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058514+0000", 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8887999999999998e-05 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058526+0000", 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1683e-05 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059537+0000", 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001011907 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059564+0000", 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.220 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.794391802, 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048595679999999999, 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394452+0000", 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397390+0000", 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002937977 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397401+0000", 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1172000000000001e-05 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397407+0000", 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397452+0000", 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4705e-05 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397467+0000", 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5622000000000001e-05 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399273+0000", 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001805823 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399310+0000", 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7512999999999998e-05 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1363 4.2 e43/31 4:4ba9f189:::smithi20438004-14:head v 43'50, mlcod=43'48)", 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.143259368999999, 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021379849999999998, 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.225 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045585+0000", 2024-09-13T11:33:11.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.604e-06 2024-09-13T11:33:11.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045958+0000", 2024-09-13T11:33:11.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000372672 2024-09-13T11:33:11.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045978+0000", 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0262999999999999e-05 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045991+0000", 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2811e-05 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046079+0000", 2024-09-13T11:33:11.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8667999999999999e-05 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046103+0000", 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4131e-05 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047664+0000", 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001560098 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047721+0000", 2024-09-13T11:33:11.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7738000000000002e-05 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1410 4.4 e43/41 4:2451796c:::smithi20438004-17:head v 43'44, mlcod=43'42)", 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.060286861000002, 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022888489999999999, 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128557+0000", 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3939999999999999e-06 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129014+0000", 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045637100000000001 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129032+0000", 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8685000000000002e-05 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129042+0000", 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129100+0000", 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8238e-05 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129119+0000", 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8258e-05 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130809+0000", 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001690045 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130845+0000", 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5991000000000002e-05 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.157696954999999, 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071247150000000002, 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.234 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031147+0000", 2024-09-13T11:33:11.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.579e-06 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036431+0000", 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00528394 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036449+0000", 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7329e-05 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036457+0000", 2024-09-13T11:33:11.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036529+0000", 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1506000000000003e-05 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036551+0000", 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2154000000000002e-05 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038234+0000", 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001683192 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038270+0000", 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6408000000000001e-05 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1468 4.4 e44/41 4:2b094039:::smithi20438004-40:head v 44'48, mlcod=43'46)", 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:33:11.238 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.119848040999997, 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020495919999999998, 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068995+0000", 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.068996+0000", 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4649999999999999e-06 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069273+0000", 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027673599999999997 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069292+0000", 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9202999999999999e-05 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069302+0000", 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069367+0000", 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5112999999999998e-05 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.069393+0000", 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5867000000000002e-05 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.070994+0000", 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016011980000000001 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.071044+0000", 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0421999999999999e-05 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1679 4.4 4.9c0290d4 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.129639251, 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47635528100000002, 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:11.243 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1679 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:11.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059205+0000", 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.812e-06 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059211+0000", 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1469999999999998e-06 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059222+0000", 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0560000000000001e-05 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059233+0000", 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0591000000000001e-05 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059286+0000", 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519e-05 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059293+0000", 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4690000000000002e-06 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535521+0000", 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47622830100000002 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535559+0000", 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7882000000000001e-05 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:11.248 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.786391621, 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002436952, 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:11.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402453+0000", 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.632e-06 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403268+0000", 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081553400000000003 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403289+0000", 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0353e-05 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403298+0000", 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4760000000000004e-06 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403369+0000", 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0858000000000005e-05 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403397+0000", 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8518e-05 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404841+0000", 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001443228 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404888+0000", 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7352999999999999e-05 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.252 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.780865224999999, 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003287178, 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:11.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407979+0000", 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409585+0000", 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016060479999999999 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409598+0000", 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3027000000000001e-05 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409606+0000", 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2190000000000004e-06 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409676+0000", 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9791999999999999e-05 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409707+0000", 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1692000000000001e-05 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411223+0000", 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001515705 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411265+0000", 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1649000000000001e-05 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1705 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'62, mlcod=0'0)", 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.773260825000001, 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003138161, 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.257 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415583+0000", 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9699999999999995e-07 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416900+0000", 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001317251 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416926+0000", 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5622e-05 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416942+0000", 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6741000000000001e-05 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417014+0000", 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1298999999999996e-05 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417042+0000", 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.851e-05 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418673+0000", 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016310949999999999 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418720+0000", 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6845999999999998e-05 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1706 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'63, mlcod=45'61)", 2024-09-13T11:33:11.261 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 25.767928628, 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003090597, 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:11.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420915+0000", 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-06 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422306+0000", 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001390753 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.263 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422317+0000", 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422322+0000", 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4360000000000001e-06 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422379+0000", 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7003000000000003e-05 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422405+0000", 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6424e-05 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.423958+0000", 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015526680000000001 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.424005+0000", 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6619000000000002e-05 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.265 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1919 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'63, mlcod=53'61)", 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.222728650000001, 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002758908, 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:11.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966116+0000", 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0370000000000001e-06 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967135+0000", 2024-09-13T11:33:11.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010185260000000001 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967149+0000", 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3903999999999999e-05 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967157+0000", 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.405e-06 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967255+0000", 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8441000000000002e-05 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967280+0000", 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5079000000000002e-05 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968828+0000", 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001547936 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968873+0000", 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4579999999999997e-05 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.214777927, 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028733999999999999, 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.270 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974066+0000", 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:33:11.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975332+0000", 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012659349999999999 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975345+0000", 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3402e-05 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975351+0000", 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4450000000000001e-06 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975438+0000", 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6687999999999994e-05 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975457+0000", 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8913e-05 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976901+0000", 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014437180000000001 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976938+0000", 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7441999999999998e-05 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.269644161999999, 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022722770000000001, 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919200+0000", 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9000000000000005e-07 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919903+0000", 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070307200000000001 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919915+0000", 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2482e-05 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919922+0000", 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5459999999999997e-06 2024-09-13T11:33:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919988+0000", 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6377999999999994e-05 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.920027+0000", 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8649999999999998e-05 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921419+0000", 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013924530000000001 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921471+0000", 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1706000000000003e-05 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2077 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'67, mlcod=53'65)", 2024-09-13T11:33:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 19.261501531, 2024-09-13T11:33:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026288430000000001, 2024-09-13T11:33:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927342+0000", 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928463+0000", 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011212640000000001 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928476+0000", 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3147e-05 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928484+0000", 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928549+0000", 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5001999999999999e-05 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928565+0000", 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6124999999999999e-05 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929914+0000", 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001348859 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929970+0000", 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5912000000000003e-05 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.285 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.285 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:11.286 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:11.286 INFO:tasks.rados.rados.0.smithi204.stdout:1318: expect (ObjNum 215 snap 0 seq_num 215) 2024-09-13T11:33:11.287 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1313017 2024-09-13T11:33:11.287 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:11.284+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:11.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:11.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:11.387 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:11.386+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:11.390 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:11.390 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:11.390 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:11.390 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:11.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:11.445 INFO:tasks.rados.rados.0.smithi204.stdout:1321: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-13T11:33:11.488 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:11.485+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:11.495 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:11.496 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:11.496 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:11.503 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:11.503 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:11.503 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:11.503 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:11.503 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:11.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:11.513 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:11.514 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:11.514 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:11.531 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:11.531 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:11.547 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:11.548 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:11.564 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:11.582 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:11.588 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:11.587+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:11.599 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:11.599 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:11.615 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:11.615 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:11.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:206 3.1 e38/30 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.521131369000003, 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012415890000000001, 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092192+0000", 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.63e-07 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100358+0000", 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081664059999999993 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100384+0000", 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5287e-05 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100401+0000", 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7744000000000002e-05 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100478+0000", 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6769999999999999e-05 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100508+0000", 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0134e-05 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104514+0000", 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040060590000000002 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104607+0000", 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3027000000000002e-05 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:201 3.1 e38/30 3:b265d015:::smithi20438004-2 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.512886029999997, 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012387119, 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.620 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100438+0000", 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1000000000000001e-06 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108964+0000", 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085255959999999999 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108984+0000", 2024-09-13T11:33:11.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0466e-05 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108996+0000", 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2149e-05 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109097+0000", 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010042799999999999 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109121+0000", 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.455e-05 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112773+0000", 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036519780000000002 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112824+0000", 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0852e-05 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.504264558999999, 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096300529999999995, 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109060+0000", 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116086+0000", 2024-09-13T11:33:11.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070260239999999996 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116100+0000", 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4209999999999999e-05 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116109+0000", 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6569999999999992e-06 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116160+0000", 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0470000000000003e-05 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116191+0000", 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1494e-05 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118639+0000", 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024481030000000001 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118688+0000", 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8866999999999998e-05 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.497192763999998, 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010922463, 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.629 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116131+0000", 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6899999999999998e-07 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124407+0000", 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082764919999999999 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124422+0000", 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5404999999999999e-05 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124432+0000", 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8910000000000007e-06 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124485+0000", 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2750999999999997e-05 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124508+0000", 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3076999999999999e-05 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127004+0000", 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024962930000000001 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127053+0000", 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8084999999999998e-05 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.633 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.483528665999998, 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091903760000000001, 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129795+0000", 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8700000000000001e-07 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135325+0000", 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055302190000000003 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135340+0000", 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135351+0000", 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1358999999999999e-05 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135418+0000", 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000002e-05 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135447+0000", 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8415000000000002e-05 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138936+0000", 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034896800000000002 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138985+0000", 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8338999999999999e-05 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.473333594000003, 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010981263, 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:11.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139990+0000", 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2e-07 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147357+0000", 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073669950000000003 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147376+0000", 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9453000000000002e-05 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147388+0000", 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1527e-05 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147472+0000", 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4630999999999997e-05 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147493+0000", 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0687e-05 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150899+0000", 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034054739999999999 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150971+0000", 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1976000000000003e-05 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.642 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.465882585999999, 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015742025999999999, 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147441+0000", 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7700000000000004e-07 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157019+0000", 2024-09-13T11:33:11.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095782090000000007 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157057+0000", 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157072+0000", 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5082999999999999e-05 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157157+0000", 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5065e-05 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157188+0000", 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0920000000000002e-05 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163133+0000", 2024-09-13T11:33:11.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059446890000000004 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163182+0000", 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9564999999999997e-05 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.455810497999998, 2024-09-13T11:33:11.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015556076, 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157514+0000", 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167722+0000", 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010208016 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167740+0000", 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7668000000000001e-05 2024-09-13T11:33:11.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167754+0000", 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4647999999999999e-05 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167831+0000", 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7229000000000002e-05 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167855+0000", 2024-09-13T11:33:11.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3467000000000001e-05 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173001+0000", 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051458160000000001 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173068+0000", 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7720999999999998e-05 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.445542005999997, 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011370243, 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3499999999999996e-07 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175562+0000", 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077811060000000003 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175578+0000", 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175587+0000", 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175668+0000", 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1224999999999996e-05 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175695+0000", 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6505000000000001e-05 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179075+0000", 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003380508 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179151+0000", 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5927000000000007e-05 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.655 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.437715345000001, 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017514140000000001, 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188284+0000", 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012675702 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188299+0000", 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5122e-05 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188308+0000", 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9819999999999997e-06 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188367+0000", 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8774999999999999e-05 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188387+0000", 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0649000000000002e-05 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193056+0000", 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046684830000000002 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193122+0000", 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5790999999999994e-05 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.424990366999999, 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015855429000000001, 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.660 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188333+0000", 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-07 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199600+0000", 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011266717000000001 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199613+0000", 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3682e-05 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199620+0000", 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1339999999999998e-06 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199676+0000", 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5689000000000002e-05 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199708+0000", 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2082999999999998e-05 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204123+0000", 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004414734 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204188+0000", 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:11.664 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.413681830000002, 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092010560000000009, 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:11.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199642+0000", 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9599999999999999e-07 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205539+0000", 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058972290000000004 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205553+0000", 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4577e-05 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205564+0000", 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0254e-05 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205630+0000", 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6057999999999997e-05 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205658+0000", 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7979000000000001e-05 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208792+0000", 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031342499999999999 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208842+0000", 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0213000000000002e-05 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:245 3.1 e38/30 3:adf25eb4:::smithi20438004-46 low_tier:head v 38'15, mlcod=38'11)", 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.407735500999998, 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011146911000000001, 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205588+0000", 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213344+0000", 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077554440000000002 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213359+0000", 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5455999999999999e-05 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213369+0000", 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9890000000000004e-06 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213453+0000", 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4153999999999994e-05 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213479+0000", 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5831999999999999e-05 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216694+0000", 2024-09-13T11:33:11.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032149990000000001 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216734+0000", 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0102999999999999e-05 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:260 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:11.673 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.393964480000001, 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011239344, 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 260 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219360+0000", 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-13T11:33:11.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219366+0000", 2024-09-13T11:33:11.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7179999999999999e-06 2024-09-13T11:33:11.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219376+0000", 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9990000000000003e-06 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219382+0000", 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8150000000000002e-06 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219428+0000", 2024-09-13T11:33:11.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5751999999999998e-05 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4119000000000001e-05 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227887+0000", 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083953989999999996 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228998+0000", 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001110764 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229290+0000", 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029145400000000002 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230489+0000", 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001199786 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230548+0000", 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8900999999999997e-05 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230598+0000", 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9391999999999997e-05 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:249 3.1 e38/30 3:81faf980:::smithi20438004-50 low_tier:head v 38'21, mlcod=38'18)", 2024-09-13T11:33:11.680 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.390688343999997, 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091134460000000007, 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:11.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.68e-07 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227843+0000", 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005207494 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227853+0000", 2024-09-13T11:33:11.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117e-05 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227860+0000", 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4880000000000001e-06 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227912+0000", 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1900000000000001e-05 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227933+0000", 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0474000000000001e-05 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231709+0000", 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037762759999999999 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231748+0000", 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9029000000000002e-05 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:278 4.2 4:4d4c8512:::smithi20438004-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.382328743999999, 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091148359999999994, 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-09-13T11:33:11.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230995+0000", 2024-09-13T11:33:11.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4200000000000004e-07 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231000+0000", 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3290000000000001e-06 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231017+0000", 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6242000000000001e-05 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231025+0000", 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231094+0000", 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9554999999999994e-05 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231171+0000", 2024-09-13T11:33:11.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7082999999999994e-05 2024-09-13T11:33:11.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:11.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236799+0000", 2024-09-13T11:33:11.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056282260000000001 2024-09-13T11:33:11.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237539+0000", 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073930700000000003 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237574+0000", 2024-09-13T11:33:11.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5191999999999998e-05 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240013+0000", 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002439185 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240070+0000", 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6650999999999997e-05 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240109+0000", 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9131999999999997e-05 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:287 4.2 4:4297cb20:::smithi20438004-38:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.378393271999997, 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013317569, 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.692 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 287 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234931+0000", 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.192e-06 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234936+0000", 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0640000000000001e-06 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234953+0000", 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7292e-05 2024-09-13T11:33:11.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234961+0000", 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235026+0000", 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5073000000000005e-05 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235111+0000", 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4635999999999994e-05 2024-09-13T11:33:11.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242149+0000", 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070384990000000001 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069074700000000002 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242936+0000", 2024-09-13T11:33:11.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5730000000000007e-05 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248138+0000", 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052018330000000003 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248197+0000", 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9512000000000002e-05 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248247+0000", 2024-09-13T11:33:11.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:310 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.364466903999997, 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095515259999999994, 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.698 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 310 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248857+0000", 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-06 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248863+0000", 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6529999999999999e-06 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248874+0000", 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.093e-05 2024-09-13T11:33:11.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248887+0000", 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3009e-05 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248932+0000", 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5445999999999998e-05 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249025+0000", 2024-09-13T11:33:11.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3098999999999998e-05 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251503+0000", 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024772629999999999 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252148+0000", 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064507900000000001 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252433+0000", 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028559100000000002 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258304+0000", 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058703100000000001 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258361+0000", 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7658999999999998e-05 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258407+0000", 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6319999999999997e-05 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:313 4.2 4:4ba9f189:::smithi20438004-14:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.363588196999999, 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013958247999999999, 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:11.704 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 313 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:11.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249736+0000", 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9999999999999996e-07 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249741+0000", 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5929999999999996e-06 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249750+0000", 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7369999999999995e-06 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249757+0000", 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5360000000000004e-06 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249806+0000", 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9018999999999999e-05 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249882+0000", 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5943000000000001e-05 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258629+0000", 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087464589999999998 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259299+0000", 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067041799999999999 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259523+0000", 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223964 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263636+0000", 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041130539999999997 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263669+0000", 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2849999999999999e-05 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263693+0000", 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3774999999999999e-05 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1437 4.2 4:4b088bf8:::smithi20438004-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.809347811000002, 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014642921, 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:11.710 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1437 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:11.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803976+0000", 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.35e-06 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803984+0000", 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2080000000000003e-06 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803995+0000", 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804001+0000", 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2870000000000001e-06 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804096+0000", 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5424e-05 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804165+0000", 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8596000000000005e-05 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.818618+0000", 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452906999999999 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:11.714 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:11.716 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:11.716 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:11.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:11.717 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:11.718 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:11.718 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:11.719 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:11.719 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:33:11.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:11.720 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:11.720 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:11.721 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:11.722 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:11.723 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:11.715+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:11.723 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:11.723 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1314: done (8 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1315: done (7 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1317: done (6 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1318: done (5 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1319: done (4 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1320: done (3 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1321: done (2 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1322: done (1 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1323: done (0 left) 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1324: set_chunk oid 6 target oid 2024-09-13T11:33:11.724 INFO:tasks.rados.rados.0.smithi204.stdout:1324: set_chunk oid 6 offset: 917504 length: 998400 target oid 80e34aaeddb218b4d39e982e468c599ffc66b1bc4b72df5d64614f55bf65e83c offset: 0 snap -1 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1325: write (excl) oid 12 current snap is 0 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1325: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1325: writing smithi20438004-12 from 601750 to 1137296 tid 1 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1325: writing smithi20438004-12 from 1929910 to 2201750 tid 2 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1326: read oid 25 snap -1 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1326: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1327: delete oid 3 current snap is 0 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1324: finishing set_chunk to oid 6 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 917504 length: 998400 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1325: finishing write tid 1 to smithi20438004-12 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1324: done (3 left) 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1327: done (2 left) 2024-09-13T11:33:11.725 INFO:tasks.rados.rados.0.smithi204.stdout:1328: read oid 44 snap -1 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1328: expect (ObjNum 223 snap 0 seq_num 223) 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1329: tier_evict oid 42 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1329: tier_evict oid 42 snap -1 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1330: set_chunk oid 18 target oid 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1330: set_chunk oid 18 offset: 1008128 length: 663040 target oid 11b3a6d289dab705f5e91f5bb6e84d9318af316208d189d427701d937d005663 offset: 0 snap -1 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1331: read oid 36 snap -1 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1331: expect (ObjNum 222 snap 0 seq_num 222) 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1329: got (22) Invalid argument 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1325: finishing write tid 2 to smithi20438004-12 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1332: read oid 27 snap -1 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1332: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1329: done (6 left) 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1333: delete oid 21 current snap is 0 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1325: finishing write tid 3 to smithi20438004-12 2024-09-13T11:33:11.726 INFO:tasks.rados.rados.0.smithi204.stdout:1325: finishing write tid 4 to smithi20438004-12 2024-09-13T11:33:11.727 INFO:tasks.rados.rados.0.smithi204.stdout:1325: oid 12 updating version 0 to 78 2024-09-13T11:33:11.727 INFO:tasks.rados.rados.0.smithi204.stdout:1325: oid 12 version 78 is already newer than 77 2024-09-13T11:33:11.727 INFO:tasks.rados.rados.0.smithi204.stdout:1325: oid 12 version 78 is already newer than 76 2024-09-13T11:33:11.727 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 78 (ObjNum 250 snap 0 seq_num 250) dirty exists 2024-09-13T11:33:11.727 INFO:tasks.rados.rados.0.smithi204.stdout:1325: left oid 12 (ObjNum 250 snap 0 seq_num 250) 2024-09-13T11:33:11.727 INFO:tasks.rados.rados.0.smithi204.stdout:1326: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-13T11:33:11.727 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:11.728 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:11.729 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:11.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:11.817 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:11.815+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:11.824 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2951024 2024-09-13T11:33:11.824 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:11.824 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:11.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:11.833 INFO:tasks.rados.rados.0.smithi204.stdout:1328: expect (ObjNum 223 snap 0 seq_num 223) 2024-09-13T11:33:11.917 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:11.916+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:11.936 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:11.936 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:11.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:11.987 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:12.018 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:12.017+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:12.048 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.048 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:12.076 INFO:tasks.rados.rados.0.smithi204.stdout:1330: finishing set_chunk to oid 18 2024-09-13T11:33:12.076 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 18 set_chunk returned error code -95 offset: 1008128 length: 663040 2024-09-13T11:33:12.076 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 0 (ObjNum 148 snap 0 seq_num 148) dirty exists 2024-09-13T11:33:12.076 INFO:tasks.rados.rados.0.smithi204.stdout:1331: expect (ObjNum 222 snap 0 seq_num 222) 2024-09-13T11:33:12.098 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2649169 2024-09-13T11:33:12.119 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:12.119+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:12.175 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.175 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:12.214 INFO:tasks.rados.rados.0.smithi204.stdout:1332: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-13T11:33:12.219 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:12.217+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:12.232 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:12.268 INFO:tasks.rados.rados.0.smithi204.stdout:1325: done (6 left) 2024-09-13T11:33:12.268 INFO:tasks.rados.rados.0.smithi204.stdout:1326: done (5 left) 2024-09-13T11:33:12.269 INFO:tasks.rados.rados.0.smithi204.stdout:1328: done (4 left) 2024-09-13T11:33:12.269 INFO:tasks.rados.rados.0.smithi204.stdout:1330: done (3 left) 2024-09-13T11:33:12.269 INFO:tasks.rados.rados.0.smithi204.stdout:1331: done (2 left) 2024-09-13T11:33:12.269 INFO:tasks.rados.rados.0.smithi204.stdout:1332: done (1 left) 2024-09-13T11:33:12.269 INFO:tasks.rados.rados.0.smithi204.stdout:1333: done (0 left) 2024-09-13T11:33:12.270 INFO:tasks.rados.rados.0.smithi204.stdout:1334: read oid 38 snap -1 2024-09-13T11:33:12.270 INFO:tasks.rados.rados.0.smithi204.stdout:1334: expect (ObjNum 227 snap 0 seq_num 227) 2024-09-13T11:33:12.270 INFO:tasks.rados.rados.0.smithi204.stdout:1335: read oid 31 snap -1 2024-09-13T11:33:12.270 INFO:tasks.rados.rados.0.smithi204.stdout:1335: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:33:12.270 INFO:tasks.rados.rados.0.smithi204.stdout:1336: read oid 16 snap -1 2024-09-13T11:33:12.270 INFO:tasks.rados.rados.0.smithi204.stdout:1336: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:33:12.270 INFO:tasks.rados.rados.0.smithi204.stdout:1337: write oid 23 current snap is 0 2024-09-13T11:33:12.271 INFO:tasks.rados.rados.0.smithi204.stdout:1337: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-13T11:33:12.272 INFO:tasks.rados.rados.0.smithi204.stdout:1337: writing smithi20438004-23 from 652171 to 1129817 tid 1 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1337: writing smithi20438004-23 from 1821709 to 2252171 tid 2 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1338: read oid 26 snap -1 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1338: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1339: read oid 33 snap -1 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1339: expect (ObjNum 215 snap 0 seq_num 215) 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1340: read oid 48 snap -1 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1340: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1341: read oid 17 snap -1 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1341: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-13T11:33:12.280 INFO:tasks.rados.rados.0.smithi204.stdout:1342: tier_evict oid 24 2024-09-13T11:33:12.281 INFO:tasks.rados.rados.0.smithi204.stdout:1342: tier_evict oid 24 snap -1 2024-09-13T11:33:12.281 INFO:tasks.rados.rados.0.smithi204.stdout:1335: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:33:12.292 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.292 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:12.345 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:12.344+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:12.396 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1488726 2024-09-13T11:33:12.405 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.405 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:12.426 INFO:tasks.rados.rados.0.smithi204.stdout:1334: expect (ObjNum 227 snap 0 seq_num 227) 2024-09-13T11:33:12.426 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:12.454 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:12.453+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:12.521 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:12.519+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:12.557 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.557 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:12.620 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3637437 2024-09-13T11:33:12.622 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:12.621+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:12.625 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:12.668 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.669 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:12.723 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:12.722+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:12.733 INFO:tasks.rados.rados.0.smithi204.stdout:1337: finishing write tid 1 to smithi20438004-23 2024-09-13T11:33:12.734 INFO:tasks.rados.rados.0.smithi204.stdout:1336: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-13T11:33:12.734 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:12.780 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.780 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:12.824 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:12.823+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:12.856 INFO:tasks.rados.rados.0.smithi204.stdout:1342: got (22) Invalid argument 2024-09-13T11:33:12.856 INFO:tasks.rados.rados.0.smithi204.stdout:1339: expect (ObjNum 215 snap 0 seq_num 215) 2024-09-13T11:33:12.891 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:12.892 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:12.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:12.925 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:12.922+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:12.980 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 850752 2024-09-13T11:33:13.002 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.003 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:13.038 INFO:tasks.rados.rados.0.smithi204.stdout:1341: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-13T11:33:13.039 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:13.054 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:13.053+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:13.114 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.114 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.123 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:13.127 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:13.126+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:13.195 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2815876 2024-09-13T11:33:13.228 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.229 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.232 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:13.229+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:13.237 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:13.281 INFO:tasks.rados.rados.0.smithi204.stdout:1340: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-13T11:33:13.351 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:13.328+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:13.395 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.395 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:13.430 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:13.429+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:13.432 INFO:tasks.rados.rados.0.smithi204.stdout:1338: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-13T11:33:13.432 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:13.505 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.505 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:13.523 INFO:tasks.rados.rados.0.smithi204.stdout:1337: finishing write tid 2 to smithi20438004-23 2024-09-13T11:33:13.523 INFO:tasks.rados.rados.0.smithi204.stdout:1337: finishing write tid 3 to smithi20438004-23 2024-09-13T11:33:13.523 INFO:tasks.rados.rados.0.smithi204.stdout:1337: finishing write tid 4 to smithi20438004-23 2024-09-13T11:33:13.523 INFO:tasks.rados.rados.0.smithi204.stdout:1337: oid 23 updating version 0 to 79 2024-09-13T11:33:13.523 INFO:tasks.rados.rados.0.smithi204.stdout:1337: oid 23 version 79 is already newer than 78 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1337: oid 23 updating version 79 to 80 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 80 (ObjNum 253 snap 0 seq_num 253) dirty exists 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1337: left oid 23 (ObjNum 253 snap 0 seq_num 253) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1334: done (8 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1335: done (7 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1336: done (6 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1337: done (5 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1338: done (4 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1339: done (3 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1340: done (2 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1341: done (1 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1342: done (0 left) 2024-09-13T11:33:13.524 INFO:tasks.rados.rados.0.smithi204.stdout:1343: read oid 1 snap -1 2024-09-13T11:33:13.525 INFO:tasks.rados.rados.0.smithi204.stdout:1343: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:13.525 INFO:tasks.rados.rados.0.smithi204.stdout:1344: set_chunk oid 26 target oid 2024-09-13T11:33:13.535 INFO:tasks.rados.rados.0.smithi204.stdout:1344: set_chunk oid 26 offset: 131072 length: 1391104 target oid 2fe429c0fa52c94fd8a9bc933e369484a85717b4d3349c5eb58827a189dd42ff offset: 0 snap -1 2024-09-13T11:33:13.535 INFO:tasks.rados.rados.0.smithi204.stdout:1345: write oid 9 current snap is 0 2024-09-13T11:33:13.535 INFO:tasks.rados.rados.0.smithi204.stdout:1345: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-13T11:33:13.535 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:13.533+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:13.541 INFO:tasks.rados.rados.0.smithi204.stdout:1345: writing smithi20438004-9 from 668978 to 1221873 tid 1 2024-09-13T11:33:13.547 INFO:tasks.rados.rados.0.smithi204.stdout:1345: writing smithi20438004-9 from 1763838 to 2268978 tid 2 2024-09-13T11:33:13.547 INFO:tasks.rados.rados.0.smithi204.stdout:1346: read oid 50 snap -1 2024-09-13T11:33:13.547 INFO:tasks.rados.rados.0.smithi204.stdout:1346: expect deleted 2024-09-13T11:33:13.547 INFO:tasks.rados.rados.0.smithi204.stdout:1347: write oid 7 current snap is 0 2024-09-13T11:33:13.547 INFO:tasks.rados.rados.0.smithi204.stdout:1347: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-13T11:33:13.554 INFO:tasks.rados.rados.0.smithi204.stdout:1347: writing smithi20438004-7 from 685785 to 1313929 tid 1 2024-09-13T11:33:13.560 INFO:tasks.rados.rados.0.smithi204.stdout:1347: writing smithi20438004-7 from 1822320 to 2285785 tid 2 2024-09-13T11:33:13.560 INFO:tasks.rados.rados.0.smithi204.stdout:1348: read oid 13 snap -1 2024-09-13T11:33:13.560 INFO:tasks.rados.rados.0.smithi204.stdout:1348: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:13.560 INFO:tasks.rados.rados.0.smithi204.stdout:1349: read oid 11 snap -1 2024-09-13T11:33:13.561 INFO:tasks.rados.rados.0.smithi204.stdout:1349: expect (ObjNum 237 snap 0 seq_num 237) 2024-09-13T11:33:13.561 INFO:tasks.rados.rados.0.smithi204.stdout:1350: read oid 19 snap -1 2024-09-13T11:33:13.561 INFO:tasks.rados.rados.0.smithi204.stdout:1350: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:33:13.561 INFO:tasks.rados.rados.0.smithi204.stdout:1351: write oid 14 current snap is 0 2024-09-13T11:33:13.561 INFO:tasks.rados.rados.0.smithi204.stdout:1343: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:13.627 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:13.632 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:13.631+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:13.645 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.645 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.654 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:13.664 INFO:tasks.rados.rados.0.smithi204.stdout:1344: finishing set_chunk to oid 26 2024-09-13T11:33:13.665 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 56 (ObjNum 248 snap 0 seq_num 248) dirty exists 2024-09-13T11:33:13.665 INFO:tasks.rados.rados.0.smithi204.stdout:1345: finishing write tid 1 to smithi20438004-9 2024-09-13T11:33:13.665 INFO:tasks.rados.rados.0.smithi204.stdout:1345: finishing write tid 2 to smithi20438004-9 2024-09-13T11:33:13.665 INFO:tasks.rados.rados.0.smithi204.stdout:1345: finishing write tid 3 to smithi20438004-9 2024-09-13T11:33:13.665 INFO:tasks.rados.rados.0.smithi204.stdout:1345: finishing write tid 4 to smithi20438004-9 2024-09-13T11:33:13.665 INFO:tasks.rados.rados.0.smithi204.stdout:1345: oid 9 updating version 0 to 44 2024-09-13T11:33:13.665 INFO:tasks.rados.rados.0.smithi204.stdout:1345: oid 9 updating version 44 to 46 2024-09-13T11:33:13.666 INFO:tasks.rados.rados.0.smithi204.stdout:1345: oid 9 version 46 is already newer than 45 2024-09-13T11:33:13.666 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 46 (ObjNum 254 snap 0 seq_num 254) dirty exists 2024-09-13T11:33:13.666 INFO:tasks.rados.rados.0.smithi204.stdout:1345: left oid 9 (ObjNum 254 snap 0 seq_num 254) 2024-09-13T11:33:13.666 INFO:tasks.rados.rados.0.smithi204.stdout:1347: finishing write tid 1 to smithi20438004-7 2024-09-13T11:33:13.666 INFO:tasks.rados.rados.0.smithi204.stdout:1351: seq_num 256 ranges {702592=703393,1880802=421790} 2024-09-13T11:33:13.671 INFO:tasks.rados.rados.0.smithi204.stdout:1351: writing smithi20438004-14 from 702592 to 1405985 tid 1 2024-09-13T11:33:13.677 INFO:tasks.rados.rados.0.smithi204.stdout:1351: writing smithi20438004-14 from 1880802 to 2302592 tid 2 2024-09-13T11:33:13.678 INFO:tasks.rados.rados.0.smithi204.stdout:1343: done (8 left) 2024-09-13T11:33:13.678 INFO:tasks.rados.rados.0.smithi204.stdout:1344: done (7 left) 2024-09-13T11:33:13.678 INFO:tasks.rados.rados.0.smithi204.stdout:1345: done (6 left) 2024-09-13T11:33:13.678 INFO:tasks.rados.rados.0.smithi204.stdout:1346: done (5 left) 2024-09-13T11:33:13.678 INFO:tasks.rados.rados.0.smithi204.stdout:1352: write (excl) oid 5 current snap is 0 2024-09-13T11:33:13.678 INFO:tasks.rados.rados.0.smithi204.stdout:1347: finishing write tid 2 to smithi20438004-7 2024-09-13T11:33:13.679 INFO:tasks.rados.rados.0.smithi204.stdout:1347: finishing write tid 3 to smithi20438004-7 2024-09-13T11:33:13.679 INFO:tasks.rados.rados.0.smithi204.stdout:1347: finishing write tid 4 to smithi20438004-7 2024-09-13T11:33:13.679 INFO:tasks.rados.rados.0.smithi204.stdout:1347: oid 7 updating version 0 to 68 2024-09-13T11:33:13.679 INFO:tasks.rados.rados.0.smithi204.stdout:1347: oid 7 updating version 68 to 69 2024-09-13T11:33:13.679 INFO:tasks.rados.rados.0.smithi204.stdout:1347: oid 7 updating version 69 to 70 2024-09-13T11:33:13.679 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 70 (ObjNum 255 snap 0 seq_num 255) dirty exists 2024-09-13T11:33:13.680 INFO:tasks.rados.rados.0.smithi204.stdout:1347: left oid 7 (ObjNum 255 snap 0 seq_num 255) 2024-09-13T11:33:13.680 INFO:tasks.rados.rados.0.smithi204.stdout:1352: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-13T11:33:13.685 INFO:tasks.rados.rados.0.smithi204.stdout:1352: writing smithi20438004-5 from 719399 to 1498041 tid 1 2024-09-13T11:33:13.692 INFO:tasks.rados.rados.0.smithi204.stdout:1352: writing smithi20438004-5 from 1939284 to 2319399 tid 2 2024-09-13T11:33:13.692 INFO:tasks.rados.rados.0.smithi204.stdout:1347: done (5 left) 2024-09-13T11:33:13.692 INFO:tasks.rados.rados.0.smithi204.stdout:1353: set_chunk oid 49 target oid 2024-09-13T11:33:13.692 INFO:tasks.rados.rados.0.smithi204.stdout:1353: set_chunk oid 49 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:13.692 INFO:tasks.rados.rados.0.smithi204.stdout:1354: read oid 45 snap -1 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1354: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1355: read oid 35 snap -1 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1355: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1356: read oid 40 snap -1 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1356: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1357: read oid 20 snap -1 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1357: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1358: set_chunk oid 47 target oid 2024-09-13T11:33:13.693 INFO:tasks.rados.rados.0.smithi204.stdout:1358: set_chunk oid 47 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1359: tier_promote oid 32 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1360: tier_flush oid 25 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1360: tier_flush oid 25 snap -1 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1361: tier_flush oid 22 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1361: tier_flush oid 22 snap -1 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1362: read oid 3 snap -1 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1362: expect deleted 2024-09-13T11:33:13.694 INFO:tasks.rados.rados.0.smithi204.stdout:1363: set_chunk oid 43 target oid 2024-09-13T11:33:13.707 INFO:tasks.rados.rados.0.smithi204.stdout:1363: set_chunk oid 43 offset: 2130944 length: 1513472 target oid 7c71280847c69742c25e255f063a7bf96a1765ba1f097b602155cc64f066d7c0 offset: 0 snap -1 2024-09-13T11:33:13.708 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:13.708 INFO:tasks.rados.rados.0.smithi204.stdout:1349: expect (ObjNum 237 snap 0 seq_num 237) 2024-09-13T11:33:13.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:33:13.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:13.767 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:13.766+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:13.770 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2542064 2024-09-13T11:33:13.783 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.783 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.791 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:13.834 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:13.833+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:13.912 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:13.913 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:13.935 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:13.934+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:13.936 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:13.936 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:13.944 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:13.946 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:13.947 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:13.947 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:13.949 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3627034 2024-09-13T11:33:13.969 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:13.986 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:13.986 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:14.003 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:14.003 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:14.020 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:14.021 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:14.036 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:14.035+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:14.038 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:14.038 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:14.050 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:14.050 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:14.056 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:14.056 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:14.059 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:14.073 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:14.074 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:14.090 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:14.090 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:14.106 INFO:tasks.rados.rados.0.smithi204.stdout:1348: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:14.107 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:14.108 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:14.108 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:14.126 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:14.126 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:14.129 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1856511 2024-09-13T11:33:14.137 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:14.135+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:14.143 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:14.143 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:14.160 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:14.161 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:14.163 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:14.163 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:14.163 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:14.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:14.172 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:14.176 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:14.237 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:14.235+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:14.278 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:14.278 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:14.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:14.278 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.174881008, 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016706373, 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:14.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.99e-07 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104537+0000", 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034575700000000001 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104552+0000", 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5475999999999999e-05 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104566+0000", 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3766e-05 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104623+0000", 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7046999999999998e-05 2024-09-13T11:33:14.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104650+0000", 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6786000000000001e-05 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117752+0000", 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013102387 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117785+0000", 2024-09-13T11:33:14.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2741999999999999e-05 2024-09-13T11:33:14.284 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.284 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.284 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.170921681999999, 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025912691000000002, 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105039+0000", 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7100000000000001e-07 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108050+0000", 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030115910000000001 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108061+0000", 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0512e-05 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108069+0000", 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4479999999999994e-06 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108126+0000", 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7024999999999997e-05 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108165+0000", 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8701000000000001e-05 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130916+0000", 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022750829 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130951+0000", 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4913999999999998e-05 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.289 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:203 3.1 e38/30 3:970ba768:::smithi20438004-4 low_tier:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.167715188999999, 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031827052000000002, 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:14.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108245+0000", 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1800000000000005e-07 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111202+0000", 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029571440000000001 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111221+0000", 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8887000000000001e-05 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111248+0000", 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6458e-05 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111707+0000", 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045935799999999999 2024-09-13T11:33:14.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111729+0000", 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140022+0000", 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028292571999999998 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140071+0000", 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9669000000000002e-05 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.164691638999997, 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028930938999999999, 2024-09-13T11:33:14.295 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111269+0000", 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5700000000000002e-07 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113854+0000", 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025854340000000002 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113866+0000", 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1884e-05 2024-09-13T11:33:14.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113873+0000", 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116175+0000", 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023019 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116218+0000", 2024-09-13T11:33:14.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2796000000000002e-05 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140165+0000", 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023947718999999999 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140199+0000", 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3596000000000002e-05 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:218 3.1 e38/30 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.154828731000002, 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025863384999999999, 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:14.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121132+0000", 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3260000000000001e-06 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124734+0000", 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036019379999999998 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124748+0000", 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913999999999999e-05 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124760+0000", 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1786e-05 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124849+0000", 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8882999999999996e-05 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124874+0000", 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5466999999999999e-05 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146945+0000", 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022070837999999999 2024-09-13T11:33:14.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146994+0000", 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9233000000000001e-05 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.147895548000001, 2024-09-13T11:33:14.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020872166000000001, 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128066+0000", 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.86e-06 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131456+0000", 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033901360000000002 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131467+0000", 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0806e-05 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131474+0000", 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131524+0000", 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9262999999999999e-05 2024-09-13T11:33:14.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131554+0000", 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0148999999999999e-05 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148883+0000", 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017329445999999998 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148936+0000", 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2879e-05 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.144457349, 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017580887, 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.309 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131503+0000", 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133995+0000", 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024920630000000001 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134006+0000", 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0696000000000001e-05 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134012+0000", 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136541+0000", 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025286150000000001 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136563+0000", 2024-09-13T11:33:14.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2531000000000002e-05 2024-09-13T11:33:14.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149031+0000", 2024-09-13T11:33:14.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012467951 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149083+0000", 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2040999999999999e-05 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.315 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:14.315 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:14.315 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.118248770999998, 2024-09-13T11:33:14.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022274233000000001, 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157712+0000", 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3689999999999999e-06 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161974+0000", 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042619160000000001 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161990+0000", 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5506e-05 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161998+0000", 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7520000000000002e-06 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162103+0000", 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104415 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162126+0000", 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3643e-05 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179932+0000", 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017805476000000001 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179985+0000", 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3155999999999997e-05 2024-09-13T11:33:14.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.113923538000002, 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018684003000000001, 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162037+0000", 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.083e-06 2024-09-13T11:33:14.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166161+0000", 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041238799999999999 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166179+0000", 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8329999999999999e-05 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166190+0000", 2024-09-13T11:33:14.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0822000000000001e-05 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166302+0000", 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011233700000000001 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166328+0000", 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5185999999999999e-05 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180646+0000", 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014318261000000001 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180720+0000", 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.324 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.109744536000001, 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014657148, 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1300000000000001e-07 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169754+0000", 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035389779999999999 2024-09-13T11:33:14.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169772+0000", 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8020999999999999e-05 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0037999999999999e-05 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169847+0000", 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4695000000000003e-05 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169865+0000", 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7365999999999999e-05 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180834+0000", 2024-09-13T11:33:14.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010969016 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180872+0000", 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8621000000000002e-05 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.097833383000001, 2024-09-13T11:33:14.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018123856000000001, 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:14.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178128+0000", 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181360+0000", 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032324699999999999 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181371+0000", 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0978000000000001e-05 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181378+0000", 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.861e-06 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181431+0000", 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3173000000000001e-05 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181455+0000", 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4236999999999999e-05 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196213+0000", 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01475807 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196250+0000", 2024-09-13T11:33:14.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6636e-05 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.094549825999998, 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016397140000000001, 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:14.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181411+0000", 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7399999999999991e-07 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184468+0000", 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030575759999999998 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184481+0000", 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3072e-05 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184491+0000", 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184550+0000", 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8251999999999997e-05 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184575+0000", 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689000000000001e-05 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197720+0000", 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013144708999999999 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197807+0000", 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6799999999999996e-05 2024-09-13T11:33:14.342 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:281 4.a 4:56ef4fd4:::smithi20438004-32:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.043511113999998, 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018358335999999999, 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.343 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232449+0000", 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7499999999999998e-07 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232457+0000", 2024-09-13T11:33:14.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3610000000000003e-06 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232470+0000", 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3230999999999999e-05 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232478+0000", 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5989999999999996e-06 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232541+0000", 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3717e-05 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232619+0000", 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248106+0000", 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015486616 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248991+0000", 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088509199999999998 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249363+0000", 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037176599999999999 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250719+0000", 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001356255 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250765+0000", 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6053000000000001e-05 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250807+0000", 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1675000000000003e-05 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1696 4.b 4:da92e835:::smithi20438004-50:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.007359299000001, 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016782392, 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1696 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268602+0000", 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268608+0000", 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7629999999999997e-06 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268620+0000", 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1950000000000001e-05 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268632+0000", 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2330999999999999e-05 2024-09-13T11:33:14.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269210+0000", 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057761900000000003 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269221+0000", 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.168e-05 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277483+0000", 2024-09-13T11:33:14.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0082621180000000006 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277641+0000", 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000157365 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285383+0000", 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00774195 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1726 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.007290842, 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015213417999999999, 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1726 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.58e-07 2024-09-13T11:33:14.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268673+0000", 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7809999999999999e-06 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268679+0000", 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7729999999999996e-06 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268684+0000", 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269275+0000", 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059124500000000001 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269281+0000", 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2779999999999999e-06 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281010+0000", 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011728762 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281114+0000", 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103872 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283882+0000", 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027680539999999998 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.359 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1727 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 28.007269095000002, 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016698689999999999, 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1727 2024-09-13T11:33:14.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268691+0000", 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6100000000000001e-07 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268694+0000", 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.672e-06 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268699+0000", 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1819999999999997e-06 2024-09-13T11:33:14.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268703+0000", 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269302+0000", 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059838199999999995 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269307+0000", 2024-09-13T11:33:14.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3170000000000001e-06 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284218+0000", 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014911308999999999 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284330+0000", 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111715 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285389+0000", 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010592729999999999 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1774 4.a 4:532cb07c:::smithi20438004-26:head [sparse-read 39610~1695928] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273521+0000", 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 27.002438468000001, 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011941719, 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.365 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1774 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273521+0000", 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273521+0000", 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273522+0000", 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.44e-07 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273525+0000", 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0989999999999999e-06 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273531+0000", 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.657e-06 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273537+0000", 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.981e-06 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273582+0000", 2024-09-13T11:33:14.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.422e-05 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273589+0000", 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6790000000000005e-06 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285103+0000", 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011513399000000001 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285226+0000", 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012324699999999999 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285463+0000", 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023679300000000001 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1775 4.a 4:532cb07c:::smithi20438004-26:head [sparse-read 1243799~869921] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273601+0000", 2024-09-13T11:33:14.370 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 27.002358134000001, 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012147214999999999, 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1775 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273601+0000", 2024-09-13T11:33:14.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273601+0000", 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273602+0000", 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6700000000000003e-07 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273606+0000", 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6480000000000001e-06 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273612+0000", 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6239999999999996e-06 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273618+0000", 2024-09-13T11:33:14.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7710000000000003e-06 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273659+0000", 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0837000000000002e-05 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273666+0000", 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0620000000000003e-06 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285482+0000", 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011816491 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285559+0000", 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6854000000000007e-05 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285749+0000", 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018936100000000001 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.375 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2018 4.5 4:a9b0c9bd:::smithi20438004-37:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.400301418000002, 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021048396, 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.376 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2018 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875659+0000", 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875661+0000", 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9920000000000002e-06 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875665+0000", 2024-09-13T11:33:14.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875668+0000", 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875714+0000", 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5189999999999999e-05 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875770+0000", 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6824999999999999e-05 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.896707+0000", 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020936156000000001 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2574 4.a 4:532cb07c:::smithi20438004-26:head [read 363008~2507776] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:14.380 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 12.101263868, 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013408788, 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2574 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:14.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174698+0000", 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0169999999999999e-06 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174703+0000", 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.989e-06 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174708+0000", 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5329999999999996e-06 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.185700+0000", 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010992500000000001 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.185762+0000", 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2199000000000005e-05 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.188104+0000", 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023420530000000002 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.384 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.385 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.385 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.385 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.385 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:14.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:14.387 INFO:tasks.rados.rados.0.smithi204.stdout:1350: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-13T11:33:14.387 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:14.385+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:14.389 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:14.449 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 581859 2024-09-13T11:33:14.487 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:14.486+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:14.493 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:14.493 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:14.493 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:14.493 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: finishing write tid 1 to smithi20438004-14 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: finishing write tid 2 to smithi20438004-14 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: finishing write tid 3 to smithi20438004-14 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: finishing write tid 4 to smithi20438004-14 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: oid 14 updating version 0 to 97 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: oid 14 updating version 97 to 98 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: oid 14 version 98 is already newer than 96 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 98 (ObjNum 256 snap 0 seq_num 256) dirty exists 2024-09-13T11:33:14.499 INFO:tasks.rados.rados.0.smithi204.stdout:1351: left oid 14 (ObjNum 256 snap 0 seq_num 256) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1348: done (15 left) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1349: done (14 left) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1350: done (13 left) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1351: done (12 left) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1364: read oid 17 snap -1 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1364: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1353: finishing set_chunk to oid 49 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1353: got expected ENOENT (src dne) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1352: finishing write tid 1 to smithi20438004-5 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1358: finishing set_chunk to oid 47 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1358: got expected ENOENT (src dne) 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1359: got (0) Success 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 50 (ObjNum 239 snap 0 seq_num 239) dirty exists 2024-09-13T11:33:14.500 INFO:tasks.rados.rados.0.smithi204.stdout:1352: finishing write tid 2 to smithi20438004-5 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1365: read oid 21 snap -1 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1365: expect deleted 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1353: done (13 left) 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1358: done (12 left) 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1359: done (11 left) 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1362: done (10 left) 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1366: read oid 29 snap -1 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1366: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1367: read oid 8 snap -1 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1367: expect (ObjNum 220 snap 0 seq_num 220) 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1368: write (excl) oid 4 current snap is 0 2024-09-13T11:33:14.501 INFO:tasks.rados.rados.0.smithi204.stdout:1368: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-13T11:33:14.503 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:14.506 INFO:tasks.rados.rados.0.smithi204.stdout:1368: writing smithi20438004-4 from 736206 to 1190097 tid 1 2024-09-13T11:33:14.512 INFO:tasks.rados.rados.0.smithi204.stdout:1368: writing smithi20438004-4 from 1881413 to 2336206 tid 2 2024-09-13T11:33:14.512 INFO:tasks.rados.rados.0.smithi204.stdout:1369: tier_evict oid 33 2024-09-13T11:33:14.512 INFO:tasks.rados.rados.0.smithi204.stdout:1369: tier_evict oid 33 snap -1 2024-09-13T11:33:14.512 INFO:tasks.rados.rados.0.smithi204.stdout:1354: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-13T11:33:14.589 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:14.587+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:14.609 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:14.610 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:14.610 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:14.610 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:14.610 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:14.618 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:14.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:14.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:14.676 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:14.677 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-13T11:33:14.677 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.636690 2024-09-13T11:33:14.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.6366904986392443 2024-09-13T11:33:14.684 INFO:tasks.rados.rados.0.smithi204.stdout:1352: finishing write tid 3 to smithi20438004-5 2024-09-13T11:33:14.685 INFO:tasks.rados.rados.0.smithi204.stdout:1352: finishing write tid 4 to smithi20438004-5 2024-09-13T11:33:14.685 INFO:tasks.rados.rados.0.smithi204.stdout:1352: oid 5 updating version 0 to 67 2024-09-13T11:33:14.685 INFO:tasks.rados.rados.0.smithi204.stdout:1352: oid 5 version 67 is already newer than 66 2024-09-13T11:33:14.685 INFO:tasks.rados.rados.0.smithi204.stdout:1352: oid 5 updating version 67 to 68 2024-09-13T11:33:14.685 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 68 (ObjNum 257 snap 0 seq_num 257) dirty exists 2024-09-13T11:33:14.685 INFO:tasks.rados.rados.0.smithi204.stdout:1352: left oid 5 (ObjNum 257 snap 0 seq_num 257) 2024-09-13T11:33:14.685 INFO:tasks.rados.rados.0.smithi204.stdout:1357: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:14.686 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:33:14.690 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:14.688+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:14.763 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.669958485000002, 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.764 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:33:14.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.767 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.767 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:33:14.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:33:14.767 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:33:14.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:14.769 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.659683917999999, 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:33:14.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:33:14.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:33:14.774 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.774 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.774 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.647016016999999, 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.778 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.640406169999999, 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:33:14.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:14.784 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.63202914, 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:14.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.612767093000002, 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010735344000000001, 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.789 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148571+0000", 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154390+0000", 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058191809999999997 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154404+0000", 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4584e-05 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154415+0000", 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0661e-05 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154478+0000", 2024-09-13T11:33:14.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2528999999999997e-05 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154502+0000", 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4460999999999998e-05 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159243+0000", 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047404259999999998 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159305+0000", 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2582999999999994e-05 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.606897961000001, 2024-09-13T11:33:14.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:14.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:33:14.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.597996682999998, 2024-09-13T11:33:14.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:33:14.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.803 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.590398960000002, 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:33:14.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:33:14.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.584995030999998, 2024-09-13T11:33:14.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.812 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.572371341, 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:33:14.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.558191891, 2024-09-13T11:33:14.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:33:14.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:33:14.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.549973723000001, 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:33:14.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:33:14.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:14.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.519150320000001, 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.828 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:33:14.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:33:14.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.372435533000001, 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:33:14.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 38.667263480999999, 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.839 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:33:14.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 37.919311110999999, 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013151014000000001, 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842045+0000", 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865e-05 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848182+0000", 2024-09-13T11:33:14.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061372370000000002 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848205+0000", 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.298e-05 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848218+0000", 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.348e-05 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848300+0000", 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1766000000000003e-05 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848327+0000", 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6446999999999999e-05 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855119+0000", 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067926490000000004 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855177+0000", 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7590000000000003e-05 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1458 4.e 4:7984cf84:::smithi20438004-46:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 37.733699059000003, 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012793834, 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.848 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1458 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027640+0000", 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.046e-06 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027651+0000", 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.102e-05 2024-09-13T11:33:14.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027669+0000", 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027681+0000", 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2177999999999999e-05 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027762+0000", 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0331e-05 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027868+0000", 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106443 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.040432+0000", 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012563632999999999 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.852 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1787 4.e 4:7022fc62:::smithi20438004-6:head [read 188928~1265152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 27.487606368000002, 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013752110999999999, 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.853 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1787 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273731+0000", 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273734+0000", 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3510000000000002e-06 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273741+0000", 2024-09-13T11:33:14.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273787+0000", 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6198000000000001e-05 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274432+0000", 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064478300000000001 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274461+0000", 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9051999999999999e-05 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281751+0000", 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007289963 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281984+0000", 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023257099999999999 2024-09-13T11:33:14.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.287483+0000", 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054989829999999998 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2762 4.e 4:7984cf84:::smithi20438004-46:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e66)", 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 7.86514895, 2024-09-13T11:33:14.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022335361000000001, 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2762 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896189+0000", 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3519999999999999e-06 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896196+0000", 2024-09-13T11:33:14.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896210+0000", 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3439e-05 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:14.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896217+0000", 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5889999999999997e-06 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896278+0000", 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1072000000000001e-05 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896338+0000", 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9998999999999999e-05 2024-09-13T11:33:14.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.918523+0000", 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022185099999999999 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:14.863 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:14.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:14.865 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2831775 2024-09-13T11:33:14.865 INFO:tasks.rados.rados.0.smithi204.stdout:1355: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:14.866 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:14.862+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:14.965 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:14.963+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:14.974 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:14.974 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:14.974 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:14.974 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:14.974 INFO:tasks.rados.rados.0.smithi204.stdout:1356: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:14.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:15.054 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2445604 2024-09-13T11:33:15.066 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:15.065+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:15.089 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:15.089 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:15.089 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:15.090 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:15.090 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:15.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:1363: finishing set_chunk to oid 43 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 43 set_chunk returned error code -95 offset: 2130944 length: 1513472 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 0 (ObjNum 228 snap 0 seq_num 228) dirty exists 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:1360: got (0) Success 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 0 (ObjNum 193 snap 0 seq_num 193) dirty exists 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:1361: got (0) Success 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 219 snap 0 seq_num 219) dirty exists 2024-09-13T11:33:15.104 INFO:tasks.rados.rados.0.smithi204.stdout:1352: done (13 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1354: done (12 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1355: done (11 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1356: done (10 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1357: done (9 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1360: done (8 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1361: done (7 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1363: done (6 left) 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1370: write oid 24 current snap is 0 2024-09-13T11:33:15.105 INFO:tasks.rados.rados.0.smithi204.stdout:1370: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-13T11:33:15.110 INFO:tasks.rados.rados.0.smithi204.stdout:1370: writing smithi20438004-24 from 753013 to 1398506 tid 1 2024-09-13T11:33:15.115 INFO:tasks.rados.rados.0.smithi204.stdout:1370: writing smithi20438004-24 from 2056248 to 2353013 tid 2 2024-09-13T11:33:15.115 INFO:tasks.rados.rados.0.smithi204.stdout:1371: set_chunk oid 38 target oid 2024-09-13T11:33:15.126 INFO:tasks.rados.rados.0.smithi204.stdout:1371: set_chunk oid 38 offset: 936448 length: 938496 target oid 56c8d704c98ca500b9ce1ad1b8c507b11c28528b88d459c96f63a5c824198f0f offset: 0 snap -1 2024-09-13T11:33:15.127 INFO:tasks.rados.rados.0.smithi204.stdout:1372: read oid 1 snap -1 2024-09-13T11:33:15.127 INFO:tasks.rados.rados.0.smithi204.stdout:1372: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:15.127 INFO:tasks.rados.rados.0.smithi204.stdout:1373: read oid 20 snap -1 2024-09-13T11:33:15.127 INFO:tasks.rados.rados.0.smithi204.stdout:1373: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:15.127 INFO:tasks.rados.rados.0.smithi204.stdout:1374: write oid 48 current snap is 0 2024-09-13T11:33:15.127 INFO:tasks.rados.rados.0.smithi204.stdout:1374: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-13T11:33:15.135 INFO:tasks.rados.rados.0.smithi204.stdout:1374: writing smithi20438004-48 from 769820 to 1490562 tid 1 2024-09-13T11:33:15.140 INFO:tasks.rados.rados.0.smithi204.stdout:1374: writing smithi20438004-48 from 2114730 to 2369820 tid 2 2024-09-13T11:33:15.140 INFO:tasks.rados.rados.0.smithi204.stdout:1375: write oid 7 current snap is 0 2024-09-13T11:33:15.140 INFO:tasks.rados.rados.0.smithi204.stdout:1375: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-13T11:33:15.148 INFO:tasks.rados.rados.0.smithi204.stdout:1375: writing smithi20438004-7 from 786627 to 1582618 tid 1 2024-09-13T11:33:15.153 INFO:tasks.rados.rados.0.smithi204.stdout:1375: writing smithi20438004-7 from 2173212 to 2386627 tid 2 2024-09-13T11:33:15.153 INFO:tasks.rados.rados.0.smithi204.stdout:1376: write (excl) oid 44 current snap is 0 2024-09-13T11:33:15.153 INFO:tasks.rados.rados.0.smithi204.stdout:1364: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-13T11:33:15.166 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:15.165+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:15.212 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:15.212 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:15.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:15.212 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:15.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.212 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'23)", 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.429933645999995, 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005604375, 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779708+0000", 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779709+0000", 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.63e-07 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779721+0000", 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.204e-05 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779728+0000", 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4150000000000002e-06 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779732+0000", 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6839999999999998e-06 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779761+0000", 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9405999999999999e-05 2024-09-13T11:33:15.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.779773+0000", 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1398e-05 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785264+0000", 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054913089999999998 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:04.785313+0000", 2024-09-13T11:33:15.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8659999999999998e-05 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.472462148000005, 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:33:15.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'105, mlcod=24'100)", 2024-09-13T11:33:15.221 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.471898545000002, 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061709499999999997, 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737744+0000", 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7700000000000004e-07 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737749+0000", 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737756+0000", 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7649999999999996e-06 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737760+0000", 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6650000000000001e-06 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742613+0000", 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048528909999999998 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742629+0000", 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6481e-05 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743902+0000", 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012723630000000001 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743914+0000", 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2663999999999999e-05 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.225 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.732536521, 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:33:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:15.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:33:15.232 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.727477962999998, 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:33:15.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:33:15.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:33:15.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.727054140999996, 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:33:15.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.727036097000003, 2024-09-13T11:33:15.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:33:15.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:33:15.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.010795324999997, 2024-09-13T11:33:15.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:33:15.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.256 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.996954535999997, 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:33:15.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:33:15.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:33:15.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.897986562, 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.263 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:33:15.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:33:15.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.b 51/50 [PushOp(4:da92e835:::smithi20438004-50:head, version: 38'19, data_included: [2616802~223548], data_size: 223548, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:da92e835:::smithi20438004-50:head@38'19, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2616802, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.897176614999999, 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055783109999999999, 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312465+0000", 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312466+0000", 2024-09-13T11:33:15.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7499999999999999e-07 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312658+0000", 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019212100000000001 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312681+0000", 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3059999999999999e-05 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312694+0000", 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2408999999999999e-05 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312765+0000", 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1755000000000007e-05 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.312775+0000", 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2879999999999998e-06 2024-09-13T11:33:15.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.318043+0000", 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052689030000000001 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1751 4.9 4:9441befd:::smithi20438004-8:head [write 420208~530210,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.824073781999999, 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060468199999999996, 2024-09-13T11:33:15.273 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1751 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385569+0000", 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.426e-06 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386470+0000", 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090069800000000002 2024-09-13T11:33:15.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386485+0000", 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5225000000000001e-05 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386495+0000", 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386574+0000", 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8911000000000004e-05 2024-09-13T11:33:15.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386698+0000", 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012372800000000001 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386913+0000", 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021553499999999999 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.388673+0000", 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017596210000000001 2024-09-13T11:33:15.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.389582+0000", 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090943100000000004 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391260+0000", 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00167782 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391570+0000", 2024-09-13T11:33:15.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030968499999999997 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391615+0000", 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5151999999999997e-05 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1752 4.9 4:9441befd:::smithi20438004-8:head [write 1527054~766050] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.816277625000001, 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062506259999999996, 2024-09-13T11:33:15.279 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1752 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393365+0000", 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0720000000000001e-06 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394108+0000", 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074250399999999995 2024-09-13T11:33:15.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394120+0000", 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1990999999999999e-05 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394125+0000", 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7270000000000004e-06 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394182+0000", 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6181000000000003e-05 2024-09-13T11:33:15.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394248+0000", 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6848999999999996e-05 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394333+0000", 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4970999999999997e-05 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396066+0000", 2024-09-13T11:33:15.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001732546 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397394+0000", 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013282680000000001 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399415+0000", 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002021063 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399585+0000", 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016999699999999999 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399615+0000", 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9456999999999999e-05 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1753 4.9 4:9441befd:::smithi20438004-8:head [setxattr _header (55) in=7b,truncate 2420208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:33:15.285 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.815500537999998, 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055775579999999998, 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1753 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:33:15.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394141+0000", 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394142+0000", 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7999999999999995e-07 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394147+0000", 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2120000000000002e-06 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394154+0000", 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6980000000000001e-06 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394160+0000", 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2269999999999998e-06 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395493+0000", 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00133299 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395574+0000", 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0944000000000003e-05 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395686+0000", 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112049 2024-09-13T11:33:15.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396457+0000", 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077122900000000001 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397581+0000", 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011240989999999999 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399655+0000", 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002074046 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399684+0000", 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8924999999999998e-05 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399719+0000", 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4459000000000003e-05 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.291 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1754 4.9 4:9441befd:::smithi20438004-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.815470061999999, 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057025110000000004, 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.292 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1754 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2300000000000001e-07 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394175+0000", 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5189999999999999e-06 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394181+0000", 2024-09-13T11:33:15.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6089999999999997e-06 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394187+0000", 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7400000000000001e-06 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395928+0000", 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017413470000000001 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:15.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395983+0000", 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399771+0000", 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003787786 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399814+0000", 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2755e-05 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399874+0000", 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0350999999999999e-05 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1959 4.e e53/51 4:7984cf84:::smithi20438004-46:head v 53'41, mlcod=52'38)", 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.021225293000001, 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057232109999999997, 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0599999999999999e-07 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188422+0000", 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9409999999999998e-06 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188434+0000", 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2002e-05 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188441+0000", 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0090000000000001e-06 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188712+0000", 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027109600000000002 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188724+0000", 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1790999999999999e-05 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194092+0000", 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053680450000000001 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194140+0000", 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7521000000000001e-05 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2036 4.e e54/51 4:76a15ccf:::smithi20438004-1:head v 54'46, mlcod=54'44)", 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.631676788, 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056625620000000003, 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577966+0000", 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4200000000000005e-07 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578870+0000", 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090414100000000002 2024-09-13T11:33:15.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578880+0000", 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0501e-05 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578886+0000", 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.356e-06 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578950+0000", 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:33:15.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578968+0000", 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7935999999999999e-05 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583569+0000", 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046012680000000004 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583628+0000", 2024-09-13T11:33:15.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8473000000000001e-05 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2293 4.9 4:9441befd:::smithi20438004-8:head [write 1586353~512385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.937768848000001, 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066015969999999998, 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.306 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2293 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271874+0000", 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.497e-06 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272359+0000", 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048419700000000001 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272399+0000", 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0754000000000003e-05 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272410+0000", 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272470+0000", 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0603999999999998e-05 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272575+0000", 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104901 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272731+0000", 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015523300000000001 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.274392+0000", 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016616070000000001 2024-09-13T11:33:15.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.276887+0000", 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024952360000000001 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278386+0000", 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014986540000000001 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278439+0000", 2024-09-13T11:33:15.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2624999999999998e-05 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278475+0000", 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5812999999999997e-05 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2494 4.d e63/31 4:b15c1843:::smithi20438004-33:head v 63'65, mlcod=63'62)", 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:15.312 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.560555139, 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064248049999999996, 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649088+0000", 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.78e-07 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649096+0000", 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3790000000000006e-06 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649108+0000", 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1751e-05 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649116+0000", 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3850000000000002e-06 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649307+0000", 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019067199999999999 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649325+0000", 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7859e-05 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655462+0000", 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061369980000000003 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655511+0000", 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9883000000000003e-05 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.316 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.4.22:41 3.1 e63/51 3:b93f8350:::9968f6d703223c53b55081382c31f0a72d88ed4f88670426e6b8ab5df371e41b:head v 63'230, mlcod=63'227)", 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.108185654, 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063303989999999996, 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101457+0000", 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101463+0000", 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7529999999999998e-06 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101473+0000", 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0154e-05 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101483+0000", 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101542+0000", 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9132999999999998e-05 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101560+0000", 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7156e-05 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107733+0000", 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061735740000000002 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107787+0000", 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3389e-05 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.321 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:15.322 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:15.323 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:15.320+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:15.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.3 primary-affinity to 0.63669 (8417262) 2024-09-13T11:33:15.324 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:15.409 INFO:tasks.rados.rados.0.smithi204.stdout:1366: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:15.423 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:15.422+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:15.472 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:15.472 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:15.472 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:15.472 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:15.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:15.495 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1783031 2024-09-13T11:33:15.525 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:15.522+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:15.530 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:15.589 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:15.589 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:15.589 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:15.589 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:15.589 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:15.595 INFO:tasks.rados.rados.0.smithi204.stdout:1369: got (22) Invalid argument 2024-09-13T11:33:15.595 INFO:tasks.rados.rados.0.smithi204.stdout:1368: finishing write tid 1 to smithi20438004-4 2024-09-13T11:33:15.595 INFO:tasks.rados.rados.0.smithi204.stdout:1376: seq_num 262 ranges {403434=471240,1315341=400756} 2024-09-13T11:33:15.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:15.600 INFO:tasks.rados.rados.0.smithi204.stdout:1376: writing smithi20438004-44 from 403434 to 874674 tid 1 2024-09-13T11:33:15.605 INFO:tasks.rados.rados.0.smithi204.stdout:1376: writing smithi20438004-44 from 1315341 to 1716097 tid 2 2024-09-13T11:33:15.605 INFO:tasks.rados.rados.0.smithi204.stdout:1364: done (12 left) 2024-09-13T11:33:15.605 INFO:tasks.rados.rados.0.smithi204.stdout:1365: done (11 left) 2024-09-13T11:33:15.605 INFO:tasks.rados.rados.0.smithi204.stdout:1366: done (10 left) 2024-09-13T11:33:15.605 INFO:tasks.rados.rados.0.smithi204.stdout:1369: done (9 left) 2024-09-13T11:33:15.605 INFO:tasks.rados.rados.0.smithi204.stdout:1377: set_chunk oid 39 target oid 2024-09-13T11:33:15.624 INFO:tasks.rados.rados.0.smithi204.stdout:1377: set_chunk oid 39 offset: 67072 length: 2022912 target oid 35d746116e015b742d2355e1d921a21f28a06eaf655b8cfcf0782553e2cf9a5a offset: 0 snap -1 2024-09-13T11:33:15.624 INFO:tasks.rados.rados.0.smithi204.stdout:1378: set_chunk oid 2 target oid 2024-09-13T11:33:15.626 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:15.624+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:15.635 INFO:tasks.rados.rados.0.smithi204.stdout:1378: set_chunk oid 2 offset: 590336 length: 1051136 target oid 210c62c0282351a32644302fc3bbd913795193b6af1c457c894c53849f297c0f offset: 0 snap -1 2024-09-13T11:33:15.635 INFO:tasks.rados.rados.0.smithi204.stdout:1379: write oid 19 current snap is 0 2024-09-13T11:33:15.635 INFO:tasks.rados.rados.0.smithi204.stdout:1368: finishing write tid 2 to smithi20438004-4 2024-09-13T11:33:15.635 INFO:tasks.rados.rados.0.smithi204.stdout:1379: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-13T11:33:15.640 INFO:tasks.rados.rados.0.smithi204.stdout:1379: writing smithi20438004-19 from 420241 to 966730 tid 1 2024-09-13T11:33:15.645 INFO:tasks.rados.rados.0.smithi204.stdout:1379: writing smithi20438004-19 from 1373823 to 1918237 tid 2 2024-09-13T11:33:15.645 INFO:tasks.rados.rados.0.smithi204.stdout:1380: tier_evict oid 40 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:1380: tier_evict oid 40 snap -1 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:1368: finishing write tid 3 to smithi20438004-4 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:1368: finishing write tid 4 to smithi20438004-4 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:1368: oid 4 updating version 0 to 69 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:1368: oid 4 updating version 69 to 71 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:1368: oid 4 version 71 is already newer than 70 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 71 (ObjNum 258 snap 0 seq_num 258) dirty exists 2024-09-13T11:33:15.646 INFO:tasks.rados.rados.0.smithi204.stdout:1368: left oid 4 (ObjNum 258 snap 0 seq_num 258) 2024-09-13T11:33:15.647 INFO:tasks.rados.rados.0.smithi204.stdout:1367: expect (ObjNum 220 snap 0 seq_num 220) 2024-09-13T11:33:15.647 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:15.709 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.184633314999999, 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.710 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:15.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:33:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:33:15.717 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:26 4.8 4:1cf41451:::smithi20438004-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.179101903000003, 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083581760000000001, 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.718 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 26 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528195+0000", 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528196+0000", 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.048e-06 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528601+0000", 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040467299999999998 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528612+0000", 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.171e-05 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528620+0000", 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1140000000000006e-06 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532030+0000", 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034098779999999999 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532128+0000", 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7910000000000003e-05 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.532280+0000", 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015168700000000001 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533907+0000", 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016271 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533914+0000", 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2590000000000003e-06 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536475+0000", 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025609259999999998 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536517+0000", 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.163e-05 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536553+0000", 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6241000000000001e-05 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:27 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.178664771000001, 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082102310000000001, 2024-09-13T11:33:15.724 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 27 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.725 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528632+0000", 2024-09-13T11:33:15.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.51e-07 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528635+0000", 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8399999999999999e-06 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528639+0000", 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8139999999999998e-06 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528642+0000", 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.19e-06 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533286+0000", 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046437859999999996 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533371+0000", 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4900000000000004e-05 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533475+0000", 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104285 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534168+0000", 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069284800000000003 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.534400+0000", 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023182699999999999 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536780+0000", 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023795880000000002 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536809+0000", 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9156e-05 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536842+0000", 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3546e-05 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:28 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.178639801999999, 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083368629999999999, 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.731 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 28 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:15.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528657+0000", 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528659+0000", 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528664+0000", 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.42e-06 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.528675+0000", 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1447999999999999e-05 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533704+0000", 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050284769999999999 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533777+0000", 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3675000000000002e-05 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536883+0000", 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031062820000000001 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536929+0000", 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5237000000000001e-05 2024-09-13T11:33:15.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536994+0000", 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5152000000000002e-05 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.495695173999998, 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:33:15.736 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:33:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.742 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.487352975999997, 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:33:15.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:33:15.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.749 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.466994972999998, 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.750 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:15.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:33:15.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:33:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:33:15.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:33:15.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.459194214, 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.759 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:33:15.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:33:15.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:15.765 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.909687406000003, 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:33:15.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:797 4.8 4:1cf41451:::smithi20438004-7:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.896324843000002, 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086937119999999993, 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 797 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810974+0000", 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7010000000000001e-06 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811111+0000", 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013712699999999999 2024-09-13T11:33:15.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811123+0000", 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1854e-05 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811131+0000", 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3979999999999999e-06 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813131+0000", 2024-09-13T11:33:15.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019997740000000002 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813239+0000", 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108378 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813421+0000", 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018227099999999999 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814991+0000", 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015690599999999999 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816465+0000", 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001474784 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819575+0000", 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031099769999999999 2024-09-13T11:33:15.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819626+0000", 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0609999999999998e-05 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819666+0000", 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9777999999999998e-05 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:798 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.896151217000003, 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087603679999999993, 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 798 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.779 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:15.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811150+0000", 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811157+0000", 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811163+0000", 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813998+0000", 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002835382 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814069+0000", 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1099000000000005e-05 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814212+0000", 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142647 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816475+0000", 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022629799999999999 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816520+0000", 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5099999999999998e-05 2024-09-13T11:33:15.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819739+0000", 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032186570000000002 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819820+0000", 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1527999999999996e-05 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819906+0000", 2024-09-13T11:33:15.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5845000000000002e-05 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:799 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.896109877000001, 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089993910000000007, 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.785 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 799 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:15.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811188+0000", 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811192+0000", 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.566e-06 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811202+0000", 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4790000000000002e-06 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811209+0000", 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814715+0000", 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035061150000000002 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814798+0000", 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2854000000000003e-05 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819962+0000", 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051643419999999997 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820067+0000", 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010525 2024-09-13T11:33:15.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820186+0000", 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118886 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:805 4.8 4:1d367c64:::smithi20438004-45:head [read 0~2739508,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.896032536999996, 2024-09-13T11:33:15.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082479100000000007, 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 805 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811264+0000", 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811265+0000", 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5400000000000002e-07 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811266+0000", 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.939e-06 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811270+0000", 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4319999999999999e-06 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811274+0000", 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4150000000000001e-06 2024-09-13T11:33:15.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818160+0000", 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068854700000000003 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.818409+0000", 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024941299999999998 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819512+0000", 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011029869999999999 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 44.947498871000001, 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:33:15.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.802 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1485 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~3966419,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.597031774999998, 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01527358, 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1485 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2300000000000002e-07 2024-09-13T11:33:15.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110269+0000", 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110274+0000", 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2349999999999999e-06 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110278+0000", 2024-09-13T11:33:15.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6100000000000002e-06 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117718+0000", 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074403120000000001 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117821+0000", 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103501 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.125539+0000", 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077171590000000003 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1636 4.8 4:1d367c64:::smithi20438004-45:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.706864607999997, 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091050230000000003, 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.807 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1636 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:15.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000433+0000", 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1200000000000003e-07 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000436+0000", 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000443+0000", 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8619999999999997e-06 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000447+0000", 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8800000000000001e-06 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.001976+0000", 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001529774 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.002101+0000", 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124824 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009537+0000", 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074360850000000003 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1637 4.8 4:1d367c64:::smithi20438004-45:head [sparse-read 0~2368645] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:15.811 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.706842078999998, 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010163363, 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1637 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000457+0000", 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000460+0000", 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3720000000000001e-06 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000463+0000", 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.225e-06 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009568+0000", 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091047630000000001 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009626+0000", 2024-09-13T11:33:15.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8142e-05 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.010618+0000", 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099181000000000009 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1749 4.8 4:1d367c64:::smithi20438004-45:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.919595912999998, 2024-09-13T11:33:15.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49694346099999998, 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1749 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:15.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787703+0000", 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6840000000000001e-06 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787708+0000", 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787721+0000", 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2405000000000001e-05 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787728+0000", 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.391e-06 2024-09-13T11:33:15.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787780+0000", 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1897000000000001e-05 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787845+0000", 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5184000000000005e-05 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284644+0000", 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49679929099999998 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2310 4.18 e60/59 4:1cf41451:::smithi20438004-7:head v 60'49, mlcod=0'0)", 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.422600241000001, 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075680649999999997, 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.821 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284697+0000", 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.284698+0000", 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.043e-06 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285649+0000", 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095142200000000005 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285665+0000", 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6317999999999999e-05 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285674+0000", 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1969999999999996e-06 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285731+0000", 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7611000000000001e-05 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.285747+0000", 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6211000000000001e-05 2024-09-13T11:33:15.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.292214+0000", 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064668670000000003 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.292265+0000", 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0396000000000003e-05 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2525 4.14 4:2c957d44:::smithi20438004-22:head [write 1782380~696402] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.092343+0000", 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.614953464999999, 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084113729999999998, 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2525 2024-09-13T11:33:15.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.092343+0000", 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.092343+0000", 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.092346+0000", 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0139999999999999e-06 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093044+0000", 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069733399999999997 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093057+0000", 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3134e-05 2024-09-13T11:33:15.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093065+0000", 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1990000000000006e-06 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093131+0000", 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6416999999999998e-05 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093238+0000", 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000106109 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.093370+0000", 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013242899999999999 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.096417+0000", 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003046815 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.098025+0000", 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016082609999999999 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.100477+0000", 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002452105 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.100692+0000", 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000214861 2024-09-13T11:33:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.100755+0000", 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2694999999999995e-05 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:15.832 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:15.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:15.834 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:15.833+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:15.882 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2388070 2024-09-13T11:33:15.915 INFO:tasks.rados.rados.0.smithi204.stdout:1370: finishing write tid 1 to smithi20438004-24 2024-09-13T11:33:15.915 INFO:tasks.rados.rados.0.smithi204.stdout:1370: finishing write tid 2 to smithi20438004-24 2024-09-13T11:33:15.915 INFO:tasks.rados.rados.0.smithi204.stdout:1370: finishing write tid 3 to smithi20438004-24 2024-09-13T11:33:15.915 INFO:tasks.rados.rados.0.smithi204.stdout:1370: finishing write tid 4 to smithi20438004-24 2024-09-13T11:33:15.915 INFO:tasks.rados.rados.0.smithi204.stdout:1370: oid 24 updating version 0 to 101 2024-09-13T11:33:15.915 INFO:tasks.rados.rados.0.smithi204.stdout:1370: oid 24 version 101 is already newer than 100 2024-09-13T11:33:15.916 INFO:tasks.rados.rados.0.smithi204.stdout:1370: oid 24 version 101 is already newer than 99 2024-09-13T11:33:15.916 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 101 (ObjNum 259 snap 0 seq_num 259) dirty exists 2024-09-13T11:33:15.916 INFO:tasks.rados.rados.0.smithi204.stdout:1370: left oid 24 (ObjNum 259 snap 0 seq_num 259) 2024-09-13T11:33:15.916 INFO:tasks.rados.rados.0.smithi204.stdout:1372: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:15.934 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:15.933+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:15.961 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1362829 2024-09-13T11:33:15.981 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:15.982 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:15.982 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:15.983 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 618846 2024-09-13T11:33:15.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:16.009 INFO:tasks.rados.rados.0.smithi204.stdout:1373: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:16.074 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:16.073+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:16.136 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:16.135+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:16.138 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:16.138 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:16.138 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:16.138 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:16.138 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:16.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:16.156 INFO:tasks.rados.rados.0.smithi204.stdout:1371: finishing set_chunk to oid 38 2024-09-13T11:33:16.156 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 102 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-09-13T11:33:16.156 INFO:tasks.rados.rados.0.smithi204.stdout:1374: finishing write tid 1 to smithi20438004-48 2024-09-13T11:33:16.156 INFO:tasks.rados.rados.0.smithi204.stdout:1374: finishing write tid 2 to smithi20438004-48 2024-09-13T11:33:16.156 INFO:tasks.rados.rados.0.smithi204.stdout:1374: finishing write tid 3 to smithi20438004-48 2024-09-13T11:33:16.156 INFO:tasks.rados.rados.0.smithi204.stdout:1374: finishing write tid 4 to smithi20438004-48 2024-09-13T11:33:16.156 INFO:tasks.rados.rados.0.smithi204.stdout:1374: oid 48 updating version 0 to 67 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1374: oid 48 updating version 67 to 69 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1374: oid 48 version 69 is already newer than 68 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 69 (ObjNum 260 snap 0 seq_num 260) dirty exists 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1374: left oid 48 (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: finishing write tid 1 to smithi20438004-7 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: finishing write tid 2 to smithi20438004-7 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: finishing write tid 3 to smithi20438004-7 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: finishing write tid 4 to smithi20438004-7 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: oid 7 updating version 0 to 72 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: oid 7 version 72 is already newer than 71 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: oid 7 updating version 72 to 73 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 73 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1375: left oid 7 (ObjNum 261 snap 0 seq_num 261) 2024-09-13T11:33:16.157 INFO:tasks.rados.rados.0.smithi204.stdout:1376: finishing write tid 1 to smithi20438004-44 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1376: finishing write tid 2 to smithi20438004-44 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1367: done (12 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1368: done (11 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1370: done (10 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1371: done (9 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1372: done (8 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1373: done (7 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1374: done (6 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1375: done (5 left) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1381: read oid 35 snap -1 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1381: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1382: write (excl) oid 8 current snap is 0 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1376: finishing write tid 3 to smithi20438004-44 2024-09-13T11:33:16.158 INFO:tasks.rados.rados.0.smithi204.stdout:1382: seq_num 264 ranges {437048=621738,1832305=404425} 2024-09-13T11:33:16.161 INFO:tasks.rados.rados.0.smithi204.stdout:1382: writing smithi20438004-8 from 437048 to 1058786 tid 1 2024-09-13T11:33:16.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1382: writing smithi20438004-8 from 1832305 to 2236730 tid 2 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1383: delete oid 33 current snap is 0 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1376: finishing write tid 4 to smithi20438004-44 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1376: oid 44 updating version 0 to 51 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1376: oid 44 updating version 51 to 53 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1376: oid 44 version 53 is already newer than 52 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 53 (ObjNum 262 snap 0 seq_num 262) dirty exists 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1376: left oid 44 (ObjNum 262 snap 0 seq_num 262) 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:1377: finishing set_chunk to oid 39 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 67072 length: 2022912 2024-09-13T11:33:16.170 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 247 snap 0 seq_num 247) dirty exists 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1378: finishing set_chunk to oid 2 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 590336 length: 1051136 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1380: got (22) Invalid argument 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: finishing write tid 1 to smithi20438004-19 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: finishing write tid 2 to smithi20438004-19 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: finishing write tid 3 to smithi20438004-19 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: finishing write tid 4 to smithi20438004-19 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: oid 19 updating version 0 to 92 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: oid 19 updating version 92 to 93 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: oid 19 version 93 is already newer than 91 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 93 (ObjNum 263 snap 0 seq_num 263) dirty exists 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1379: left oid 19 (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:16.171 INFO:tasks.rados.rados.0.smithi204.stdout:1381: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:16.197 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3046439 2024-09-13T11:33:16.236 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:16.234+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:16.266 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:16.301 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.896663781000001, 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026017369999999998, 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402439+0000", 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.402441+0000", 2024-09-13T11:33:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.508e-06 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403463+0000", 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010217970000000001 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403475+0000", 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2498e-05 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403483+0000", 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4590000000000001e-06 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403531+0000", 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-05 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.403552+0000", 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0270000000000001e-05 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.404996+0000", 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001443915 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.405041+0000", 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5540000000000001e-05 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.890898790000001, 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028959070000000001, 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408205+0000", 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409400+0000", 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011945110000000001 2024-09-13T11:33:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409410+0000", 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0159999999999999e-05 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409416+0000", 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409467+0000", 2024-09-13T11:33:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1270999999999998e-05 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409489+0000", 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1682999999999999e-05 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411056+0000", 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015677569999999999 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411100+0000", 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4036999999999999e-05 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1728 4.0 e50/49 4:04aeeab9:::smithi20438004-5:head v 50'45, mlcod=0'0)", 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.854843448, 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002646874, 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444260+0000", 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.444261+0000", 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0920000000000001e-06 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445429+0000", 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00116847 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445444+0000", 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4487999999999999e-05 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445454+0000", 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8349999999999999e-06 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445509+0000", 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5154000000000001e-05 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.445525+0000", 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6654e-05 2024-09-13T11:33:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.446871+0000", 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001345534 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.446907+0000", 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5646999999999999e-05 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1904 4.5 e53/50 4:a5f8f9e2:::smithi20438004-47:head v 53'37, mlcod=52'35)", 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.366161650999999, 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025939650000000002, 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.932942+0000", 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.66e-07 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933607+0000", 2024-09-13T11:33:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00066545899999999997 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933617+0000", 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0273e-05 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933625+0000", 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5079999999999999e-06 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933679+0000", 2024-09-13T11:33:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4447999999999998e-05 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.933701+0000", 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1342999999999999e-05 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.935495+0000", 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001794604 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.935536+0000", 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0163999999999997e-05 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.324917538000001, 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026856010000000001, 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974186+0000", 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3300000000000002e-07 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974977+0000", 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079094699999999996 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.974994+0000", 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7085e-05 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975004+0000", 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3149999999999998e-06 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975070+0000", 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6167999999999994e-05 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.975097+0000", 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6855e-05 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976824+0000", 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017271039999999999 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:49.976871+0000", 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7494000000000003e-05 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.379994623999998, 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027642500000000002, 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3200000000000001e-07 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919918+0000", 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080951299999999999 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919931+0000", 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2662e-05 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919940+0000", 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3780000000000007e-06 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.920022+0000", 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2057000000000004e-05 2024-09-13T11:33:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.920044+0000", 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1863999999999999e-05 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.921822+0000", 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017779779999999999 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.921873+0000", 2024-09-13T11:33:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0565999999999997e-05 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2315 4.0 e58/56 4:001d90c2:::smithi20438004-28:head v 58'53, mlcod=57'51)", 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.179440980999999, 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031817899999999999, 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119664+0000", 2024-09-13T11:33:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8330000000000001e-06 2024-09-13T11:33:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120926+0000", 2024-09-13T11:33:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012617710000000001 2024-09-13T11:33:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120940+0000", 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4544e-05 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120952+0000", 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1676e-05 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121050+0000", 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8177999999999994e-05 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121071+0000", 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1174000000000001e-05 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122794+0000", 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00172215 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122844+0000", 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0464000000000003e-05 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2353 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'51, mlcod=0'0)", 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.306096+0000", 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.993006819000001, 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027573580000000001, 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.306096+0000", 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.306096+0000", 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.306097+0000", 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307358+0000", 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012603720000000001 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307394+0000", 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6668999999999997e-05 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307403+0000", 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4139999999999994e-06 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307471+0000", 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8088000000000003e-05 2024-09-13T11:33:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.307497+0000", 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6077000000000001e-05 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.308807+0000", 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001310121 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.308854+0000", 2024-09-13T11:33:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6826000000000002e-05 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4b088bf8:::smithi20438004-24:head, version: 60'85, data_included: [540672~892928,1630208~155648], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4b088bf8:::smithi20438004-24:head@60'85, size: 3058874, copy_subset: [(0, 3058874)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1785856, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.985913399000001, 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003437201, 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313191+0000", 2024-09-13T11:33:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.046e-06 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315519+0000", 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023280670000000001 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315551+0000", 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2440000000000001e-05 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315574+0000", 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2286e-05 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315668+0000", 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929000000000001e-05 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315679+0000", 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1e-05 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.316627+0000", 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094843300000000005 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2354 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'52, mlcod=57'50)", 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.977917249000001, 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058622930000000002, 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321187+0000", 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2200000000000003e-07 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323519+0000", 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002331891 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323532+0000", 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3494e-05 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323542+0000", 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323612+0000", 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9809999999999997e-05 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323638+0000", 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6526999999999999e-05 2024-09-13T11:33:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.326990+0000", 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033519270000000002 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327048+0000", 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8158999999999997e-05 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2355 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'53, mlcod=57'50)", 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.974868873999998, 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028910490000000001, 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324235+0000", 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6699999999999999e-07 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324282+0000", 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7437e-05 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324290+0000", 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3170000000000002e-06 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324297+0000", 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.691e-06 2024-09-13T11:33:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324944+0000", 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064681799999999996 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324977+0000", 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2591e-05 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327083+0000", 2024-09-13T11:33:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021065540000000001 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327125+0000", 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2274000000000002e-05 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2356 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'54, mlcod=57'50)", 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.974795209, 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030409059999999999, 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0100000000000001e-07 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324312+0000", 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324318+0000", 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6119999999999998e-06 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324323+0000", 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325363+0000", 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001040116 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325383+0000", 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9644999999999999e-05 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327310+0000", 2024-09-13T11:33:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001927205 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327349+0000", 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8930000000000002e-05 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2513 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'86, mlcod=0'0)", 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.233273220999999, 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034473500000000001, 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8099999999999999e-07 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066800+0000", 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096952400000000004 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066816+0000", 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6010000000000001e-05 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066826+0000", 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0679000000000001e-05 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066970+0000", 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014400900000000001 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.067020+0000", 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9595000000000002e-05 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069220+0000", 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022001640000000001 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069277+0000", 2024-09-13T11:33:16.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7188000000000002e-05 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2515 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'88, mlcod=63'86)", 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.220058613999999, 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007260702, 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4300000000000002e-07 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080302+0000", 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001256889 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080314+0000", 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.134e-05 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080319+0000", 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6699999999999999e-06 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080371+0000", 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2039000000000001e-05 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080397+0000", 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5996e-05 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086251+0000", 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058537789999999999 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086305+0000", 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4146e-05 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2516 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'89, mlcod=63'86)", 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.218682434, 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062074870000000002, 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7199999999999999e-07 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080426+0000", 2024-09-13T11:33:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080434+0000", 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2930000000000001e-06 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080443+0000", 2024-09-13T11:33:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3699999999999995e-06 2024-09-13T11:33:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081319+0000", 2024-09-13T11:33:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087608100000000002 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081346+0000", 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7282999999999999e-05 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086595+0000", 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052492759999999998 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086628+0000", 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2836999999999997e-05 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.14.0:4 4.12 e63/61 4:4b088bf8:::smithi20438004-24:head v 63'90, mlcod=63'88)", 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.123612+0000", 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.175491039000001, 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025216280000000002, 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.123612+0000", 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.123612+0000", 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.123613+0000", 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1519999999999999e-06 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.124982+0000", 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001368527 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.124997+0000", 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5407999999999999e-05 2024-09-13T11:33:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.125005+0000", 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4100000000000002e-06 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.125061+0000", 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6209999999999999e-05 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.125092+0000", 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0948999999999998e-05 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.126104+0000", 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010122659999999999 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.126134+0000", 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9706e-05 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2718 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'58, mlcod=58'56)", 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.175283681, 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039829779999999999, 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125730+0000", 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001910213 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125744+0000", 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3326000000000001e-05 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125752+0000", 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:33:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125811+0000", 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9582e-05 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125836+0000", 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4482e-05 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127747+0000", 2024-09-13T11:33:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019112599999999999 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127803+0000", 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5633000000000001e-05 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2719 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'59, mlcod=60'57)", 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.167506421000001, 2024-09-13T11:33:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032139159999999998, 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131598+0000", 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7999999999999993e-07 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132663+0000", 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010650010000000001 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132690+0000", 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6970000000000001e-05 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132703+0000", 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.36e-05 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132837+0000", 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013366699999999999 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132900+0000", 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3331999999999996e-05 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134769+0000", 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018688520000000001 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134811+0000", 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1514000000000003e-05 2024-09-13T11:33:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2720 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'60, mlcod=66'58)", 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.138205+0000", 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.160898250000001, 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025807759999999999, 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.138205+0000", 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.138205+0000", 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.138206+0000", 2024-09-13T11:33:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1599999999999999e-06 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139281+0000", 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010749760000000001 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139294+0000", 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2807e-05 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139300+0000", 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2940000000000002e-06 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139349+0000", 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8965999999999998e-05 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.139363+0000", 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3791999999999999e-05 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.140744+0000", 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013805110000000001 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.140786+0000", 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.227e-05 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3042 4.12 e71/61 4:4b088bf8:::smithi20438004-24:head v 71'99, mlcod=70'97)", 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.1828397749999999, 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036713660000000001, 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116264+0000", 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8300000000000001e-07 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117477+0000", 2024-09-13T11:33:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012129280000000001 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117486+0000", 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7100000000000002e-06 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117496+0000", 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6940000000000007e-06 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117579+0000", 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3207999999999993e-05 2024-09-13T11:33:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117606+0000", 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6622999999999999e-05 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119900+0000", 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022940899999999999 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119935+0000", 2024-09-13T11:33:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4929999999999999e-05 2024-09-13T11:33:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:16.397 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:16.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:16.400 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:16.400 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:16.401 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:16.401 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:16.402 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:16.402 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:16.402 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:16.403 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:16.403 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:16.404 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:16.404 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:16.405 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:16.397+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1382: finishing write tid 1 to smithi20438004-8 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1376: done (7 left) 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1377: done (6 left) 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1378: done (5 left) 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1379: done (4 left) 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1380: done (3 left) 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1381: done (2 left) 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1383: done (1 left) 2024-09-13T11:33:16.406 INFO:tasks.rados.rados.0.smithi204.stdout:1384: read oid 28 snap -1 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1384: expect (ObjNum 233 snap 0 seq_num 233) 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1385: read oid 41 snap -1 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1385: expect deleted 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1386: write (excl) oid 30 current snap is 0 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1382: finishing write tid 2 to smithi20438004-8 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1382: finishing write tid 3 to smithi20438004-8 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1382: finishing write tid 4 to smithi20438004-8 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1382: oid 8 updating version 0 to 33 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1382: oid 8 updating version 33 to 35 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1382: oid 8 version 35 is already newer than 34 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 35 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1382: left oid 8 (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:16.407 INFO:tasks.rados.rados.0.smithi204.stdout:1386: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1386: writing smithi20438004-30 from 470662 to 1242898 tid 1 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1386: writing smithi20438004-30 from 1832916 to 2241010 tid 2 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1382: done (3 left) 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1387: read oid 48 snap -1 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1387: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1388: write (excl) oid 44 current snap is 0 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1388: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1388: writing smithi20438004-44 from 487469 to 1051307 tid 1 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1388: writing smithi20438004-44 from 1607751 to 2159503 tid 2 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1389: read oid 5 snap -1 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1389: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1390: write (excl) oid 3 current snap is 0 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1390: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-13T11:33:16.408 INFO:tasks.rados.rados.0.smithi204.stdout:1390: writing smithi20438004-3 from 504276 to 1143363 tid 1 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1390: writing smithi20438004-3 from 1666233 to 2361643 tid 2 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1391: read oid 43 snap -1 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1391: expect (ObjNum 228 snap 0 seq_num 228) 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1392: read oid 11 snap -1 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1392: expect (ObjNum 237 snap 0 seq_num 237) 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1393: read oid 19 snap -1 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1393: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1394: read oid 18 snap -1 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1394: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1395: write oid 24 current snap is 0 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1395: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1395: writing smithi20438004-24 from 521083 to 1235419 tid 1 2024-09-13T11:33:16.409 INFO:tasks.rados.rados.0.smithi204.stdout:1395: writing smithi20438004-24 from 1724715 to 2280136 tid 2 2024-09-13T11:33:16.410 INFO:tasks.rados.rados.0.smithi204.stdout:1396: read oid 8 snap -1 2024-09-13T11:33:16.410 INFO:tasks.rados.rados.0.smithi204.stdout:1396: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:16.410 INFO:tasks.rados.rados.0.smithi204.stdout:1397: write oid 4 current snap is 0 2024-09-13T11:33:16.410 INFO:tasks.rados.rados.0.smithi204.stdout:1384: expect (ObjNum 233 snap 0 seq_num 233) 2024-09-13T11:33:16.410 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:16.411 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:16.424 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:16.424 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:16.440 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:16.440 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:16.457 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:16.457 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:16.474 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:16.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:16.490 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:16.491 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:16.491 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:16.499 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:16.498+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:16.508 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:16.508 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:16.518 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1960474 2024-09-13T11:33:16.524 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:16.524 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:16.525 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:16.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:16.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:16.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:16.542 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:16.545 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:16.546 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:16.567 INFO:tasks.rados.rados.0.smithi204.stdout:1387: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:16.601 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:16.600+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:16.606 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:16.639 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:16.639 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:16.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:16.701 INFO:tasks.rados.rados.0.smithi204.stdout:1386: finishing write tid 1 to smithi20438004-30 2024-09-13T11:33:16.701 INFO:tasks.rados.rados.0.smithi204.stdout:1386: finishing write tid 2 to smithi20438004-30 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1386: finishing write tid 3 to smithi20438004-30 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1386: finishing write tid 4 to smithi20438004-30 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1386: oid 30 updating version 0 to 72 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1386: oid 30 updating version 72 to 73 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1386: oid 30 updating version 73 to 74 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 74 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1386: left oid 30 (ObjNum 266 snap 0 seq_num 266) 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1388: finishing write tid 1 to smithi20438004-44 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1388: finishing write tid 2 to smithi20438004-44 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1388: finishing write tid 3 to smithi20438004-44 2024-09-13T11:33:16.702 INFO:tasks.rados.rados.0.smithi204.stdout:1397: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-13T11:33:16.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:33:16.703 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:16.700+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:16.708 INFO:tasks.rados.rados.0.smithi204.stdout:1397: writing smithi20438004-4 from 537890 to 1327475 tid 1 2024-09-13T11:33:16.714 INFO:tasks.rados.rados.0.smithi204.stdout:1397: writing smithi20438004-4 from 1783197 to 2482276 tid 2 2024-09-13T11:33:16.715 INFO:tasks.rados.rados.0.smithi204.stdout:1384: done (13 left) 2024-09-13T11:33:16.715 INFO:tasks.rados.rados.0.smithi204.stdout:1385: done (12 left) 2024-09-13T11:33:16.715 INFO:tasks.rados.rados.0.smithi204.stdout:1386: done (11 left) 2024-09-13T11:33:16.715 INFO:tasks.rados.rados.0.smithi204.stdout:1387: done (10 left) 2024-09-13T11:33:16.715 INFO:tasks.rados.rados.0.smithi204.stdout:1398: set_chunk oid 27 target oid 2024-09-13T11:33:16.723 INFO:tasks.rados.rados.0.smithi204.stdout:1398: set_chunk oid 27 offset: 344064 length: 718336 target oid 0a81a3e328d3cd41d708d31ff33067b9e1fc4eea6e0de9f0e9310165cb2bcd7a offset: 0 snap -1 2024-09-13T11:33:16.723 INFO:tasks.rados.rados.0.smithi204.stdout:1399: read oid 12 snap -1 2024-09-13T11:33:16.723 INFO:tasks.rados.rados.0.smithi204.stdout:1399: expect (ObjNum 250 snap 0 seq_num 250) 2024-09-13T11:33:16.723 INFO:tasks.rados.rados.0.smithi204.stdout:1400: write oid 10 current snap is 0 2024-09-13T11:33:16.724 INFO:tasks.rados.rados.0.smithi204.stdout:1388: finishing write tid 4 to smithi20438004-44 2024-09-13T11:33:16.724 INFO:tasks.rados.rados.0.smithi204.stdout:1388: oid 44 updating version 0 to 55 2024-09-13T11:33:16.724 INFO:tasks.rados.rados.0.smithi204.stdout:1388: oid 44 version 55 is already newer than 54 2024-09-13T11:33:16.724 INFO:tasks.rados.rados.0.smithi204.stdout:1388: oid 44 updating version 55 to 56 2024-09-13T11:33:16.724 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 56 (ObjNum 267 snap 0 seq_num 267) dirty exists 2024-09-13T11:33:16.724 INFO:tasks.rados.rados.0.smithi204.stdout:1388: left oid 44 (ObjNum 267 snap 0 seq_num 267) 2024-09-13T11:33:16.724 INFO:tasks.rados.rados.0.smithi204.stdout:1400: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-13T11:33:16.728 INFO:tasks.rados.rados.0.smithi204.stdout:1400: writing smithi20438004-10 from 554697 to 1019531 tid 1 2024-09-13T11:33:16.735 INFO:tasks.rados.rados.0.smithi204.stdout:1400: writing smithi20438004-10 from 1725326 to 2284416 tid 2 2024-09-13T11:33:16.735 INFO:tasks.rados.rados.0.smithi204.stdout:1388: done (12 left) 2024-09-13T11:33:16.736 INFO:tasks.rados.rados.0.smithi204.stdout:1401: write (excl) oid 17 current snap is 0 2024-09-13T11:33:16.736 INFO:tasks.rados.rados.0.smithi204.stdout:1401: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-13T11:33:16.741 INFO:tasks.rados.rados.0.smithi204.stdout:1401: writing smithi20438004-17 from 571504 to 1111587 tid 1 2024-09-13T11:33:16.749 INFO:tasks.rados.rados.0.smithi204.stdout:1401: writing smithi20438004-17 from 1783808 to 2486556 tid 2 2024-09-13T11:33:16.749 INFO:tasks.rados.rados.0.smithi204.stdout:1402: write (excl) oid 2 current snap is 0 2024-09-13T11:33:16.749 INFO:tasks.rados.rados.0.smithi204.stdout:1402: seq_num 273 ranges {588311=615332,1842290=562759} 2024-09-13T11:33:16.754 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:16.754 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:16.756 INFO:tasks.rados.rados.0.smithi204.stdout:1402: writing smithi20438004-2 from 588311 to 1203643 tid 1 2024-09-13T11:33:16.763 INFO:tasks.rados.rados.0.smithi204.stdout:1402: writing smithi20438004-2 from 1842290 to 2405049 tid 2 2024-09-13T11:33:16.763 INFO:tasks.rados.rados.0.smithi204.stdout:1403: tier_evict oid 40 2024-09-13T11:33:16.763 INFO:tasks.rados.rados.0.smithi204.stdout:1403: tier_evict oid 40 snap -1 2024-09-13T11:33:16.763 INFO:tasks.rados.rados.0.smithi204.stdout:1389: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-13T11:33:16.763 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:16.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:16.800 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1989156 2024-09-13T11:33:16.802 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:16.800+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:16.828 INFO:tasks.rados.rados.0.smithi204.stdout:1390: finishing write tid 1 to smithi20438004-3 2024-09-13T11:33:16.828 INFO:tasks.rados.rados.0.smithi204.stdout:1391: expect (ObjNum 228 snap 0 seq_num 228) 2024-09-13T11:33:16.903 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:16.902+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:16.907 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:16.908 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:16.916 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:17.004 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:17.002+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:17.019 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.019 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.027 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:17.037 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3641227 2024-09-13T11:33:17.045 INFO:tasks.rados.rados.0.smithi204.stdout:1390: finishing write tid 2 to smithi20438004-3 2024-09-13T11:33:17.045 INFO:tasks.rados.rados.0.smithi204.stdout:1390: finishing write tid 3 to smithi20438004-3 2024-09-13T11:33:17.045 INFO:tasks.rados.rados.0.smithi204.stdout:1389: done (14 left) 2024-09-13T11:33:17.045 INFO:tasks.rados.rados.0.smithi204.stdout:1391: done (13 left) 2024-09-13T11:33:17.045 INFO:tasks.rados.rados.0.smithi204.stdout:1404: write (excl) oid 14 current snap is 0 2024-09-13T11:33:17.046 INFO:tasks.rados.rados.0.smithi204.stdout:1404: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-13T11:33:17.050 INFO:tasks.rados.rados.0.smithi204.stdout:1404: writing smithi20438004-14 from 605118 to 1012052 tid 1 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1404: writing smithi20438004-14 from 1617125 to 2323542 tid 2 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1405: tier_evict oid 47 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1405: tier_evict oid 47 snap -1 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1390: finishing write tid 4 to smithi20438004-3 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1390: oid 3 updating version 0 to 36 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1390: oid 3 updating version 36 to 38 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1390: oid 3 version 38 is already newer than 37 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 38 (ObjNum 268 snap 0 seq_num 268) dirty exists 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1390: left oid 3 (ObjNum 268 snap 0 seq_num 268) 2024-09-13T11:33:17.058 INFO:tasks.rados.rados.0.smithi204.stdout:1394: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-13T11:33:17.101 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 449277 2024-09-13T11:33:17.105 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:17.104+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:17.131 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.131 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:17.193 INFO:tasks.rados.rados.0.smithi204.stdout:1393: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:17.206 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:17.205+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:17.272 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:33:17.285 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.285 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.294 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:17.311 INFO:tasks.rados.rados.0.smithi204.stdout:1392: expect (ObjNum 237 snap 0 seq_num 237) 2024-09-13T11:33:17.338 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:17.337+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:17.396 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.397 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.406 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:17.408 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:17.406+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:17.509 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:17.508+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:17.550 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.550 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:17.586 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:17.610 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:17.608+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:17.701 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.701 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:17.711 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:17.710+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:17.721 INFO:tasks.rados.rados.0.smithi204.stdout:1395: finishing write tid 1 to smithi20438004-24 2024-09-13T11:33:17.721 INFO:tasks.rados.rados.0.smithi204.stdout:1395: finishing write tid 2 to smithi20438004-24 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:1395: finishing write tid 3 to smithi20438004-24 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:1395: finishing write tid 4 to smithi20438004-24 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:1395: oid 24 updating version 0 to 103 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:1395: oid 24 updating version 103 to 104 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:1395: oid 24 version 104 is already newer than 102 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 104 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:1395: left oid 24 (ObjNum 269 snap 0 seq_num 269) 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:1396: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:17.722 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:17.812 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.812 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.813 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:17.811+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:17.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:17.914 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:17.913+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:17.963 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:17.963 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:17.969 INFO:tasks.rados.rados.0.smithi204.stdout:1397: finishing write tid 1 to smithi20438004-4 2024-09-13T11:33:17.969 INFO:tasks.rados.rados.0.smithi204.stdout:1397: finishing write tid 2 to smithi20438004-4 2024-09-13T11:33:17.969 INFO:tasks.rados.rados.0.smithi204.stdout:1397: finishing write tid 3 to smithi20438004-4 2024-09-13T11:33:17.969 INFO:tasks.rados.rados.0.smithi204.stdout:1397: finishing write tid 4 to smithi20438004-4 2024-09-13T11:33:17.969 INFO:tasks.rados.rados.0.smithi204.stdout:1397: oid 4 updating version 0 to 76 2024-09-13T11:33:17.969 INFO:tasks.rados.rados.0.smithi204.stdout:1397: oid 4 updating version 76 to 77 2024-09-13T11:33:17.969 INFO:tasks.rados.rados.0.smithi204.stdout:1397: oid 4 version 77 is already newer than 75 2024-09-13T11:33:17.970 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 77 (ObjNum 270 snap 0 seq_num 270) dirty exists 2024-09-13T11:33:17.970 INFO:tasks.rados.rados.0.smithi204.stdout:1397: left oid 4 (ObjNum 270 snap 0 seq_num 270) 2024-09-13T11:33:17.970 INFO:tasks.rados.rados.0.smithi204.stdout:1398: finishing set_chunk to oid 27 2024-09-13T11:33:17.970 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 86 (ObjNum 242 snap 0 seq_num 242) dirty exists 2024-09-13T11:33:17.970 INFO:tasks.rados.rados.0.smithi204.stdout:1399: expect (ObjNum 250 snap 0 seq_num 250) 2024-09-13T11:33:17.970 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:17.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:18.016 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:18.015+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:18.088 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:18.088 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:18.092 INFO:tasks.rados.rados.0.smithi204.stdout:1400: finishing write tid 1 to smithi20438004-10 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1400: finishing write tid 2 to smithi20438004-10 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1400: finishing write tid 3 to smithi20438004-10 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1400: finishing write tid 4 to smithi20438004-10 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1400: oid 10 updating version 0 to 114 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1400: oid 10 version 114 is already newer than 112 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1400: oid 10 version 114 is already newer than 113 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 114 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1400: left oid 10 (ObjNum 271 snap 0 seq_num 271) 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1401: finishing write tid 1 to smithi20438004-17 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1401: finishing write tid 2 to smithi20438004-17 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1401: finishing write tid 3 to smithi20438004-17 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1401: finishing write tid 4 to smithi20438004-17 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1401: oid 17 updating version 0 to 76 2024-09-13T11:33:18.093 INFO:tasks.rados.rados.0.smithi204.stdout:1401: oid 17 version 76 is already newer than 74 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1401: oid 17 version 76 is already newer than 75 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 76 (ObjNum 272 snap 0 seq_num 272) dirty exists 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1401: left oid 17 (ObjNum 272 snap 0 seq_num 272) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1402: finishing write tid 1 to smithi20438004-2 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1390: done (14 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1392: done (13 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1393: done (12 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1394: done (11 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1395: done (10 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1396: done (9 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1397: done (8 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1398: done (7 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1399: done (6 left) 2024-09-13T11:33:18.094 INFO:tasks.rados.rados.0.smithi204.stdout:1400: done (5 left) 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1401: done (4 left) 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1406: write oid 25 current snap is 0 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1403: got (22) Invalid argument 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1402: finishing write tid 2 to smithi20438004-2 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1402: finishing write tid 3 to smithi20438004-2 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1402: finishing write tid 4 to smithi20438004-2 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1402: oid 2 updating version 0 to 83 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1402: oid 2 version 83 is already newer than 82 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1402: oid 2 version 83 is already newer than 81 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 83 (ObjNum 273 snap 0 seq_num 273) dirty exists 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1402: left oid 2 (ObjNum 273 snap 0 seq_num 273) 2024-09-13T11:33:18.095 INFO:tasks.rados.rados.0.smithi204.stdout:1404: finishing write tid 1 to smithi20438004-14 2024-09-13T11:33:18.096 INFO:tasks.rados.rados.0.smithi204.stdout:1405: got (22) Invalid argument 2024-09-13T11:33:18.096 INFO:tasks.rados.rados.0.smithi204.stdout:1406: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-13T11:33:18.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:18.098 INFO:tasks.rados.rados.0.smithi204.stdout:1406: writing smithi20438004-25 from 621925 to 1104108 tid 1 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1406: writing smithi20438004-25 from 1559254 to 2125682 tid 2 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1402: done (4 left) 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1403: done (3 left) 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1405: done (2 left) 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1407: read oid 21 snap -1 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1407: expect deleted 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1408: write (excl) oid 18 current snap is 0 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1404: finishing write tid 2 to smithi20438004-14 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1404: finishing write tid 3 to smithi20438004-14 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1404: finishing write tid 4 to smithi20438004-14 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1404: oid 14 updating version 0 to 105 2024-09-13T11:33:18.104 INFO:tasks.rados.rados.0.smithi204.stdout:1404: oid 14 updating version 105 to 106 2024-09-13T11:33:18.105 INFO:tasks.rados.rados.0.smithi204.stdout:1404: oid 14 updating version 106 to 107 2024-09-13T11:33:18.105 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 107 (ObjNum 274 snap 0 seq_num 274) dirty exists 2024-09-13T11:33:18.105 INFO:tasks.rados.rados.0.smithi204.stdout:1404: left oid 14 (ObjNum 274 snap 0 seq_num 274) 2024-09-13T11:33:18.105 INFO:tasks.rados.rados.0.smithi204.stdout:1408: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-13T11:33:18.110 INFO:tasks.rados.rados.0.smithi204.stdout:1408: writing smithi20438004-18 from 638732 to 1196164 tid 1 2024-09-13T11:33:18.117 INFO:tasks.rados.rados.0.smithi204.stdout:1408: writing smithi20438004-18 from 1617736 to 2327822 tid 2 2024-09-13T11:33:18.117 INFO:tasks.rados.rados.0.smithi204.stdout:1404: done (3 left) 2024-09-13T11:33:18.117 INFO:tasks.rados.rados.0.smithi204.stdout:1409: tier_promote oid 8 2024-09-13T11:33:18.117 INFO:tasks.rados.rados.0.smithi204.stdout:1406: finishing write tid 1 to smithi20438004-25 2024-09-13T11:33:18.117 INFO:tasks.rados.rados.0.smithi204.stdout:1406: finishing write tid 2 to smithi20438004-25 2024-09-13T11:33:18.117 INFO:tasks.rados.rados.0.smithi204.stdout:1407: done (3 left) 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1410: read oid 15 snap -1 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1410: expect deleted 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1411: read oid 39 snap -1 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1411: expect (ObjNum 247 snap 0 seq_num 247) 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1412: write oid 4 current snap is 0 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1406: finishing write tid 3 to smithi20438004-25 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1406: finishing write tid 4 to smithi20438004-25 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1406: oid 25 updating version 0 to 97 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1406: oid 25 version 97 is already newer than 96 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1406: oid 25 version 97 is already newer than 95 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 97 (ObjNum 275 snap 0 seq_num 275) dirty exists 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1406: left oid 25 (ObjNum 275 snap 0 seq_num 275) 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1408: finishing write tid 1 to smithi20438004-18 2024-09-13T11:33:18.118 INFO:tasks.rados.rados.0.smithi204.stdout:1412: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-13T11:33:18.124 INFO:tasks.rados.rados.0.smithi204.stdout:1412: writing smithi20438004-4 from 655539 to 1288220 tid 1 2024-09-13T11:33:18.131 INFO:tasks.rados.rados.0.smithi204.stdout:1412: writing smithi20438004-4 from 2076218 to 2646315 tid 2 2024-09-13T11:33:18.131 INFO:tasks.rados.rados.0.smithi204.stdout:1406: done (5 left) 2024-09-13T11:33:18.131 INFO:tasks.rados.rados.0.smithi204.stdout:1413: delete oid 44 current snap is 0 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1409: got (0) Success 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 35 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1408: finishing write tid 2 to smithi20438004-18 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1408: finishing write tid 3 to smithi20438004-18 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1408: finishing write tid 4 to smithi20438004-18 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1408: oid 18 updating version 0 to 108 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1408: oid 18 version 108 is already newer than 106 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1408: oid 18 version 108 is already newer than 107 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 108 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1408: left oid 18 (ObjNum 276 snap 0 seq_num 276) 2024-09-13T11:33:18.132 INFO:tasks.rados.rados.0.smithi204.stdout:1411: expect (ObjNum 247 snap 0 seq_num 247) 2024-09-13T11:33:18.138 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:18.137+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:18.165 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:18.197 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 473162 2024-09-13T11:33:18.217 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:18.216+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: finishing write tid 1 to smithi20438004-4 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: finishing write tid 2 to smithi20438004-4 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: finishing write tid 3 to smithi20438004-4 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: finishing write tid 4 to smithi20438004-4 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: oid 4 updating version 0 to 79 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: oid 4 updating version 79 to 80 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: oid 4 version 80 is already newer than 78 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 80 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1412: left oid 4 (ObjNum 277 snap 0 seq_num 277) 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1408: done (5 left) 2024-09-13T11:33:18.227 INFO:tasks.rados.rados.0.smithi204.stdout:1409: done (4 left) 2024-09-13T11:33:18.228 INFO:tasks.rados.rados.0.smithi204.stdout:1410: done (3 left) 2024-09-13T11:33:18.228 INFO:tasks.rados.rados.0.smithi204.stdout:1411: done (2 left) 2024-09-13T11:33:18.228 INFO:tasks.rados.rados.0.smithi204.stdout:1412: done (1 left) 2024-09-13T11:33:18.228 INFO:tasks.rados.rados.0.smithi204.stdout:1413: done (0 left) 2024-09-13T11:33:18.228 INFO:tasks.rados.rados.0.smithi204.stdout:1414: write oid 28 current snap is 0 2024-09-13T11:33:18.228 INFO:tasks.rados.rados.0.smithi204.stdout:1414: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-13T11:33:18.234 INFO:tasks.rados.rados.0.smithi204.stdout:1414: writing smithi20438004-28 from 689153 to 1472332 tid 1 2024-09-13T11:33:18.240 INFO:tasks.rados.rados.0.smithi204.stdout:1414: writing smithi20438004-28 from 2076829 to 2534242 tid 2 2024-09-13T11:33:18.240 INFO:tasks.rados.rados.0.smithi204.stdout:1415: read oid 1 snap -1 2024-09-13T11:33:18.240 INFO:tasks.rados.rados.0.smithi204.stdout:1415: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:18.240 INFO:tasks.rados.rados.0.smithi204.stdout:1416: write oid 2 current snap is 0 2024-09-13T11:33:18.241 INFO:tasks.rados.rados.0.smithi204.stdout:1414: finishing write tid 1 to smithi20438004-28 2024-09-13T11:33:18.241 INFO:tasks.rados.rados.0.smithi204.stdout:1416: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-13T11:33:18.243 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:18.243 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:18.246 INFO:tasks.rados.rados.0.smithi204.stdout:1416: writing smithi20438004-2 from 705960 to 1164388 tid 1 2024-09-13T11:33:18.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1416: writing smithi20438004-2 from 1735311 to 2452735 tid 2 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1417: read oid 49 snap -1 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1417: expect deleted 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1418: read oid 38 snap -1 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1418: expect (ObjNum 227 snap 0 seq_num 227) 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1419: tier_promote oid 30 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1420: read oid 41 snap -1 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1420: expect deleted 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1421: read oid 39 snap -1 2024-09-13T11:33:18.255 INFO:tasks.rados.rados.0.smithi204.stdout:1421: expect (ObjNum 247 snap 0 seq_num 247) 2024-09-13T11:33:18.256 INFO:tasks.rados.rados.0.smithi204.stdout:1422: set_chunk oid 42 target oid 2024-09-13T11:33:18.270 INFO:tasks.rados.rados.0.smithi204.stdout:1422: set_chunk oid 42 offset: 1080832 length: 1719808 target oid b636457dbde3c7d4f7ae53d77e1805e91438706e423d10fc4daf303fdaf8be8c offset: 0 snap -1 2024-09-13T11:33:18.270 INFO:tasks.rados.rados.0.smithi204.stdout:1423: write (excl) oid 43 current snap is 0 2024-09-13T11:33:18.270 INFO:tasks.rados.rados.0.smithi204.stdout:1423: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-13T11:33:18.277 INFO:tasks.rados.rados.0.smithi204.stdout:1423: writing smithi20438004-43 from 722767 to 1256444 tid 1 2024-09-13T11:33:18.283 INFO:tasks.rados.rados.0.smithi204.stdout:1423: writing smithi20438004-43 from 1793793 to 2254875 tid 2 2024-09-13T11:33:18.283 INFO:tasks.rados.rados.0.smithi204.stdout:1424: read oid 46 snap -1 2024-09-13T11:33:18.283 INFO:tasks.rados.rados.0.smithi204.stdout:1424: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:18.283 INFO:tasks.rados.rados.0.smithi204.stdout:1425: write (excl) oid 11 current snap is 0 2024-09-13T11:33:18.283 INFO:tasks.rados.rados.0.smithi204.stdout:1415: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:18.305 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1667254 2024-09-13T11:33:18.308 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 23032 2024-09-13T11:33:18.317 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:18.316+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:18.359 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:18.359 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:18.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:18.379 INFO:tasks.rados.rados.0.smithi204.stdout:1414: finishing write tid 2 to smithi20438004-28 2024-09-13T11:33:18.379 INFO:tasks.rados.rados.0.smithi204.stdout:1414: finishing write tid 3 to smithi20438004-28 2024-09-13T11:33:18.380 INFO:tasks.rados.rados.0.smithi204.stdout:1414: finishing write tid 4 to smithi20438004-28 2024-09-13T11:33:18.380 INFO:tasks.rados.rados.0.smithi204.stdout:1414: oid 28 updating version 0 to 70 2024-09-13T11:33:18.380 INFO:tasks.rados.rados.0.smithi204.stdout:1414: oid 28 version 70 is already newer than 69 2024-09-13T11:33:18.380 INFO:tasks.rados.rados.0.smithi204.stdout:1414: oid 28 updating version 70 to 71 2024-09-13T11:33:18.380 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 71 (ObjNum 279 snap 0 seq_num 279) dirty exists 2024-09-13T11:33:18.380 INFO:tasks.rados.rados.0.smithi204.stdout:1414: left oid 28 (ObjNum 279 snap 0 seq_num 279) 2024-09-13T11:33:18.380 INFO:tasks.rados.rados.0.smithi204.stdout:1425: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-13T11:33:18.387 INFO:tasks.rados.rados.0.smithi204.stdout:1425: writing smithi20438004-11 from 739574 to 1464853 tid 1 2024-09-13T11:33:18.394 INFO:tasks.rados.rados.0.smithi204.stdout:1425: writing smithi20438004-11 from 1968628 to 2689721 tid 2 2024-09-13T11:33:18.394 INFO:tasks.rados.rados.0.smithi204.stdout:1414: done (11 left) 2024-09-13T11:33:18.394 INFO:tasks.rados.rados.0.smithi204.stdout:1415: done (10 left) 2024-09-13T11:33:18.394 INFO:tasks.rados.rados.0.smithi204.stdout:1426: read oid 40 snap -1 2024-09-13T11:33:18.394 INFO:tasks.rados.rados.0.smithi204.stdout:1426: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:18.394 INFO:tasks.rados.rados.0.smithi204.stdout:1427: write oid 18 current snap is 0 2024-09-13T11:33:18.395 INFO:tasks.rados.rados.0.smithi204.stdout:1427: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-13T11:33:18.400 INFO:tasks.rados.rados.0.smithi204.stdout:1427: writing smithi20438004-18 from 756381 to 1156909 tid 1 2024-09-13T11:33:18.408 INFO:tasks.rados.rados.0.smithi204.stdout:1427: writing smithi20438004-18 from 1910757 to 2375508 tid 2 2024-09-13T11:33:18.408 INFO:tasks.rados.rados.0.smithi204.stdout:1428: read oid 33 snap -1 2024-09-13T11:33:18.408 INFO:tasks.rados.rados.0.smithi204.stdout:1428: expect deleted 2024-09-13T11:33:18.408 INFO:tasks.rados.rados.0.smithi204.stdout:1429: read oid 17 snap -1 2024-09-13T11:33:18.408 INFO:tasks.rados.rados.0.smithi204.stdout:1429: expect (ObjNum 272 snap 0 seq_num 272) 2024-09-13T11:33:18.408 INFO:tasks.rados.rados.0.smithi204.stdout:1430: write oid 15 current snap is 0 2024-09-13T11:33:18.408 INFO:tasks.rados.rados.0.smithi204.stdout:1430: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-13T11:33:18.417 INFO:tasks.rados.rados.0.smithi204.stdout:1430: writing smithi20438004-15 from 773188 to 1248965 tid 1 2024-09-13T11:33:18.418 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:18.416+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:18.422 INFO:tasks.rados.rados.0.smithi204.stdout:1430: writing smithi20438004-15 from 1969239 to 2694001 tid 2 2024-09-13T11:33:18.423 INFO:tasks.rados.rados.0.smithi204.stdout:1431: tier_promote oid 35 2024-09-13T11:33:18.423 INFO:tasks.rados.rados.0.smithi204.stdout:1416: finishing write tid 1 to smithi20438004-2 2024-09-13T11:33:18.423 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:18.423 INFO:tasks.rados.rados.0.smithi204.stdout:1417: done (15 left) 2024-09-13T11:33:18.423 INFO:tasks.rados.rados.0.smithi204.stdout:1420: done (14 left) 2024-09-13T11:33:18.423 INFO:tasks.rados.rados.0.smithi204.stdout:1432: set_chunk oid 10 target oid 2024-09-13T11:33:18.436 INFO:tasks.rados.rados.0.smithi204.stdout:1432: set_chunk oid 10 offset: 9216 length: 1434112 target oid 3632e8b207c82dfd657ce665479db9041d6d5ae0ed7232c125bfec5a5b7cbf65 offset: 0 snap -1 2024-09-13T11:33:18.436 INFO:tasks.rados.rados.0.smithi204.stdout:1433: read oid 24 snap -1 2024-09-13T11:33:18.437 INFO:tasks.rados.rados.0.smithi204.stdout:1433: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-13T11:33:18.437 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:18.437 INFO:tasks.rados.rados.0.smithi204.stdout:1421: expect (ObjNum 247 snap 0 seq_num 247) 2024-09-13T11:33:18.470 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 357283 2024-09-13T11:33:18.473 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:18.474 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:18.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:18.519 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1894438 2024-09-13T11:33:18.519 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:18.518+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:1419: got (0) Success 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 74 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:1416: finishing write tid 2 to smithi20438004-2 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:1416: finishing write tid 3 to smithi20438004-2 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:1419: done (15 left) 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:1421: done (14 left) 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:1434: write (excl) oid 22 current snap is 0 2024-09-13T11:33:18.521 INFO:tasks.rados.rados.0.smithi204.stdout:1434: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-13T11:33:18.528 INFO:tasks.rados.rados.0.smithi204.stdout:1434: writing smithi20438004-22 from 789995 to 1341021 tid 1 2024-09-13T11:33:18.535 INFO:tasks.rados.rados.0.smithi204.stdout:1434: writing smithi20438004-22 from 2027721 to 2496141 tid 2 2024-09-13T11:33:18.535 INFO:tasks.rados.rados.0.smithi204.stdout:1435: tier_evict oid 19 2024-09-13T11:33:18.535 INFO:tasks.rados.rados.0.smithi204.stdout:1435: tier_evict oid 19 snap -1 2024-09-13T11:33:18.535 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:18.535 INFO:tasks.rados.rados.0.smithi204.stdout:1416: finishing write tid 4 to smithi20438004-2 2024-09-13T11:33:18.535 INFO:tasks.rados.rados.0.smithi204.stdout:1416: oid 2 updating version 0 to 84 2024-09-13T11:33:18.536 INFO:tasks.rados.rados.0.smithi204.stdout:1416: oid 2 updating version 84 to 86 2024-09-13T11:33:18.536 INFO:tasks.rados.rados.0.smithi204.stdout:1416: oid 2 version 86 is already newer than 85 2024-09-13T11:33:18.536 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 86 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-09-13T11:33:18.536 INFO:tasks.rados.rados.0.smithi204.stdout:1416: left oid 2 (ObjNum 280 snap 0 seq_num 280) 2024-09-13T11:33:18.536 INFO:tasks.rados.rados.0.smithi204.stdout:1418: expect (ObjNum 227 snap 0 seq_num 227) 2024-09-13T11:33:18.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:18.620 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:18.619+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:18.627 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:18.627 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:18.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:18.684 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:18.684 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:18.702 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:18.702 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:18.702 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:18.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:18.719 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:18.721 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:18.720+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:18.735 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:18.736 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:18.738 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:18.739 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:18.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:18.752 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:18.753 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:18.769 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:18.769 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:18.786 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:18.787 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:18.787 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2498099 2024-09-13T11:33:18.802 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:18.803 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:18.819 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:18.825 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:18.824+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:18.836 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:18.837 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:18.844 INFO:tasks.rados.rados.0.smithi204.stdout:1422: finishing set_chunk to oid 42 2024-09-13T11:33:18.844 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 56 (ObjNum 186 snap 0 seq_num 186) dirty exists 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: finishing write tid 1 to smithi20438004-43 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: finishing write tid 2 to smithi20438004-43 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: finishing write tid 3 to smithi20438004-43 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: finishing write tid 4 to smithi20438004-43 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: oid 43 updating version 0 to 115 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: oid 43 updating version 115 to 116 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: oid 43 updating version 116 to 117 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 117 (ObjNum 281 snap 0 seq_num 281) dirty exists 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1423: left oid 43 (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:18.845 INFO:tasks.rados.rados.0.smithi204.stdout:1424: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:18.852 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:18.852 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:18.854 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:18.854 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:18.854 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:18.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:18.871 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:18.888 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:18.888 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:18.905 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:18.905 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:18.910 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:18.926 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1453413 2024-09-13T11:33:18.927 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:18.925+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:18.966 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:18.966 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:18.966 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:18.966 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:18.967 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:18.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:19.006 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3164244 2024-09-13T11:33:19.007 INFO:tasks.rados.rados.0.smithi204.stdout:1425: finishing write tid 1 to smithi20438004-11 2024-09-13T11:33:19.007 INFO:tasks.rados.rados.0.smithi204.stdout:1425: finishing write tid 2 to smithi20438004-11 2024-09-13T11:33:19.007 INFO:tasks.rados.rados.0.smithi204.stdout:1425: finishing write tid 3 to smithi20438004-11 2024-09-13T11:33:19.007 INFO:tasks.rados.rados.0.smithi204.stdout:1425: finishing write tid 4 to smithi20438004-11 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1425: oid 11 updating version 0 to 118 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1425: oid 11 updating version 118 to 120 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1425: oid 11 version 120 is already newer than 119 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 120 (ObjNum 282 snap 0 seq_num 282) dirty exists 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1425: left oid 11 (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1416: done (15 left) 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1418: done (14 left) 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1422: done (13 left) 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1423: done (12 left) 2024-09-13T11:33:19.008 INFO:tasks.rados.rados.0.smithi204.stdout:1424: done (11 left) 2024-09-13T11:33:19.009 INFO:tasks.rados.rados.0.smithi204.stdout:1425: done (10 left) 2024-09-13T11:33:19.009 INFO:tasks.rados.rados.0.smithi204.stdout:1436: write (excl) oid 5 current snap is 0 2024-09-13T11:33:19.009 INFO:tasks.rados.rados.0.smithi204.stdout:1426: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:19.060 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:19.058+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:19.089 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2918435 2024-09-13T11:33:19.089 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:19.120 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:19.120 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 52.096961211999997, 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005153202, 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021422+0000", 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.021424+0000", 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8759999999999999e-06 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022585+0000", 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011614539999999999 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022596+0000", 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1110999999999999e-05 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022603+0000", 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4250000000000003e-06 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022647+0000", 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4487999999999998e-05 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.022662+0000", 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4817999999999999e-05 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.026542+0000", 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038797409999999999 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.026575+0000", 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3288999999999999e-05 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 51.723536310999997, 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078075499999999999, 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.125 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394848+0000", 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397492+0000", 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026441139999999999 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397513+0000", 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0381e-05 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397520+0000", 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4789999999999999e-06 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397586+0000", 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5687999999999999e-05 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397612+0000", 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5870000000000001e-05 2024-09-13T11:33:19.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402610+0000", 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049984199999999999 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402654+0000", 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4153000000000002e-05 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "pg_update_log_missing(4.e epoch 38/31 rep_tid 32 entries 38'23 (0'0) error 4:7022fc62:::smithi20438004-6:head by client.4631.0:787 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 38'20 roll_forward_to 38'20)", 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 51.718809403999998, 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057797359999999997, 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8500000000000001e-07 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399583+0000", 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4139999999999994e-06 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399602+0000", 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9176000000000001e-05 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399611+0000", 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2520000000000005e-06 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399661+0000", 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9688000000000003e-05 2024-09-13T11:33:19.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.405353+0000", 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056925209999999999 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:19.133 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 42.087131219, 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081668139999999997, 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:19.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031253+0000", 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7900000000000007e-07 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037708+0000", 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064553029999999999 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037720+0000", 2024-09-13T11:33:19.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2062000000000001e-05 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037745+0000", 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4766999999999999e-05 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037797+0000", 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2198999999999999e-05 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037815+0000", 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8178000000000001e-05 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039386+0000", 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015708110000000001 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039419+0000", 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2515000000000003e-05 2024-09-13T11:33:19.137 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1715 4.f e51/50 4:fea8c5b2:::smithi20438004-39:head v 51'67, mlcod=0'0)", 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 32.842613880999998, 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063741700000000002, 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275770+0000", 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9999999999999997e-07 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276160+0000", 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038996700000000002 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276171+0000", 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0702e-05 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276177+0000", 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7750000000000004e-06 2024-09-13T11:33:19.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276230+0000", 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2751999999999999e-05 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276252+0000", 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2055000000000001e-05 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282092+0000", 2024-09-13T11:33:19.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058396180000000004 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282143+0000", 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1601000000000003e-05 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1716 4.f e51/50 4:fea8c5b2:::smithi20438004-39:head v 51'68, mlcod=0'0)", 2024-09-13T11:33:19.142 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.277466+0000", 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 32.840916997999997, 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050982550000000003, 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277466+0000", 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277466+0000", 2024-09-13T11:33:19.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277467+0000", 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4799999999999997e-07 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277545+0000", 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7830999999999994e-05 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277554+0000", 2024-09-13T11:33:19.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.431e-06 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277560+0000", 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5310000000000001e-06 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277601+0000", 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1674000000000001e-05 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277617+0000", 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5654e-05 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282535+0000", 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049180550000000002 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282564+0000", 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9331000000000001e-05 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2285 4.e e57/51 4:7984cf84:::smithi20438004-46:head v 57'55, mlcod=56'53)", 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.887148705000001, 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055185929999999996, 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231235+0000", 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3799999999999997e-07 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232465+0000", 2024-09-13T11:33:19.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001230182 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232475+0000", 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9650000000000003e-06 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232480+0000", 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1599999999999997e-06 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232538+0000", 2024-09-13T11:33:19.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7711999999999999e-05 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232552+0000", 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.363e-05 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236731+0000", 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004179833 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236753+0000", 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1472999999999999e-05 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4d4c8512:::smithi20438004-29:head, version: 55'82, data_included: [405504~978944,1437696~69632], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4d4c8512:::smithi20438004-29:head@55'82, size: 2806769, copy_subset: [(0, 2806769)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1507328, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.784237891, 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064952710000000004, 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334146+0000", 2024-09-13T11:33:19.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9500000000000002e-07 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339863+0000", 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057170859999999997 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339880+0000", 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6705999999999999e-05 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339897+0000", 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7663999999999999e-05 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339963+0000", 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5808000000000004e-05 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339972+0000", 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6289999999999999e-06 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.340640+0000", 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066878300000000005 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2515 4.12 4:4d4c8512:::smithi20438004-29:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 18.040386655999999, 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008644934, 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.155 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2515 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077997+0000", 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5999999999999991e-07 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078584+0000", 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058694699999999999 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078595+0000", 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0808999999999999e-05 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078600+0000", 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9779999999999996e-06 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078667+0000", 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6445999999999994e-05 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078732+0000", 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5061000000000006e-05 2024-09-13T11:33:19.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078820+0000", 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8208999999999995e-05 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080991+0000", 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021715419999999998 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084271+0000", 2024-09-13T11:33:19.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032798079999999999 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086533+0000", 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022614810000000001 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086566+0000", 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2959000000000001e-05 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086641+0000", 2024-09-13T11:33:19.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5734000000000004e-05 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2516 4.12 4:4d4c8512:::smithi20438004-29:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 18.039771948999999, 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082031369999999992, 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.162 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2516 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:19.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078612+0000", 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078631+0000", 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9823999999999999e-05 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078642+0000", 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.011e-05 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078647+0000", 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.079994+0000", 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001346835 2024-09-13T11:33:19.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080073+0000", 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8721999999999996e-05 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080232+0000", 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000159698 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.081037+0000", 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080509799999999997 2024-09-13T11:33:19.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084295+0000", 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032578360000000001 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086714+0000", 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024189799999999998 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086749+0000", 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4331e-05 2024-09-13T11:33:19.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086814+0000", 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5703000000000004e-05 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2517 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 18.039728184000001, 2024-09-13T11:33:19.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083158850000000003, 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2517 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9800000000000004e-07 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078658+0000", 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1940000000000001e-06 2024-09-13T11:33:19.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078664+0000", 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1859999999999997e-06 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078668+0000", 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6820000000000002e-06 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080476+0000", 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018074160000000001 2024-09-13T11:33:19.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080525+0000", 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9036000000000003e-05 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086839+0000", 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063138650000000001 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086884+0000", 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5717999999999998e-05 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086971+0000", 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6290000000000002e-05 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2537 4.12 4:4b088bf8:::smithi20438004-24:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 17.999432799000001, 2024-09-13T11:33:19.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077763800000000003, 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2537 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118952+0000", 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0159999999999998e-06 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118976+0000", 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4227999999999999e-05 2024-09-13T11:33:19.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118987+0000", 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119009+0000", 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2232999999999999e-05 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119058+0000", 2024-09-13T11:33:19.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8427999999999998e-05 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119146+0000", 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.126727+0000", 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075806049999999998 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2746 4.e e66/51 4:727deab8:::smithi20438004-48:head v 66'59, mlcod=57'57)", 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 12.931266995, 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056203759999999998, 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188837+0000", 2024-09-13T11:33:19.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001720435 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188852+0000", 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5741000000000001e-05 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188867+0000", 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4123e-05 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188930+0000", 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.368e-05 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188949+0000", 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9004999999999998e-05 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.192684+0000", 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037346020000000001 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.192736+0000", 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2547999999999999e-05 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2890 4.f e69/59 4:fea8c5b2:::smithi20438004-39:head v 69'88, mlcod=66'85)", 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:10.980484+0000", 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.1378992589999992, 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054412949999999996, 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.182 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980484+0000", 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980484+0000", 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980485+0000", 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0100000000000001e-06 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980757+0000", 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027216700000000003 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980770+0000", 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3157e-05 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980778+0000", 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1030000000000002e-06 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980829+0000", 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1093e-05 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980858+0000", 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8861e-05 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.985876+0000", 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050179380000000004 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.985925+0000", 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8965999999999998e-05 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3033 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3663926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.6198829440000004, 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083775140000000008, 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.187 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3033 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4499999999999998e-07 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498503+0000", 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4889999999999998e-06 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498508+0000", 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7079999999999996e-06 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498511+0000", 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1879999999999998e-06 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503489+0000", 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049776849999999999 2024-09-13T11:33:19.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503558+0000", 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9128000000000006e-05 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.506878+0000", 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033199710000000001 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.191 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3042 4.12 4:4b088bf8:::smithi20438004-24:head [write 753013~645493] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.0080418550000001, 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099953939999999995, 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3042 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110345+0000", 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.563e-06 2024-09-13T11:33:19.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111084+0000", 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073883200000000003 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111101+0000", 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7039999999999999e-05 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111110+0000", 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9060000000000008e-06 2024-09-13T11:33:19.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111178+0000", 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8503000000000005e-05 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111183+0000", 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2959999999999998e-06 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112893+0000", 2024-09-13T11:33:19.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001709728 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112975+0000", 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1757999999999999e-05 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.113113+0000", 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013859500000000001 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:19.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119968+0000", 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068545869999999997 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120007+0000", 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584000000000002e-05 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120085+0000", 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8226999999999995e-05 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120188+0000", 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103243 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120337+0000", 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014853199999999999 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.198 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3043 4.12 4:4b088bf8:::smithi20438004-24:head [write 2056248~296765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.115337+0000", 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.0030456089999999, 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055019099999999996, 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3043 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115337+0000", 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115337+0000", 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115339+0000", 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:33:19.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115555+0000", 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000215837 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115570+0000", 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4893e-05 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115577+0000", 2024-09-13T11:33:19.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.109e-06 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.117911+0000", 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023347099999999998 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.118019+0000", 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107509 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:19.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.118199+0000", 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018026300000000001 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120021+0000", 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001821853 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120381+0000", 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035969399999999998 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120668+0000", 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000287414 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120748+0000", 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9374000000000003e-05 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120839+0000", 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1827e-05 2024-09-13T11:33:19.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3044 4.12 4:4b088bf8:::smithi20438004-24:head [setxattr _header (56) in=7b,truncate 2353013] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.0027911620000003, 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057109630000000003, 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3044 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:19.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5400000000000001e-07 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115596+0000", 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.002e-06 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115601+0000", 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5689999999999998e-06 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115605+0000", 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3900000000000003e-06 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119338+0000", 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037322150000000001 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119398+0000", 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0336e-05 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119499+0000", 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010123099999999999 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120393+0000", 2024-09-13T11:33:19.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00089431999999999999 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120578+0000", 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000184914 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121182+0000", 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060329600000000002 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121216+0000", 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4062e-05 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121303+0000", 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7074e-05 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3045 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:19.211 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.0027723069999999, 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060597120000000001, 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3045 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.03e-07 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115613+0000", 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.598e-06 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115616+0000", 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7500000000000001e-06 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115619+0000", 2024-09-13T11:33:19.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.089e-06 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119902+0000", 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042830250000000002 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119956+0000", 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3075999999999998e-05 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121340+0000", 2024-09-13T11:33:19.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013844669999999999 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121395+0000", 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4721e-05 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121670+0000", 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027568299999999998 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.216 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3122 4.12 4:4b088bf8:::smithi20438004-24:head [write 521083~714336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:16.374254+0000", 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.7441286900000001, 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055176000000000001, 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.217 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3122 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.374254+0000", 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.374254+0000", 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:19.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.374256+0000", 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.858e-06 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375083+0000", 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00082709999999999999 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375098+0000", 2024-09-13T11:33:19.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4369000000000001e-05 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375107+0000", 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9919999999999996e-06 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375175+0000", 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8802999999999998e-05 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375244+0000", 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8093e-05 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375366+0000", 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121946 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.377750+0000", 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023846610000000002 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379180+0000", 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001430136 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379610+0000", 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000429667 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379709+0000", 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9419999999999993e-05 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:19.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379772+0000", 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2555e-05 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:19.223 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:19.224 INFO:tasks.rados.rados.0.smithi204.stdout:1427: finishing write tid 1 to smithi20438004-18 2024-09-13T11:33:19.224 INFO:tasks.rados.rados.0.smithi204.stdout:1429: expect (ObjNum 272 snap 0 seq_num 272) 2024-09-13T11:33:19.224 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:19.225 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:19.226 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:19.223+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:19.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:33:19.325 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:19.324+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:19.329 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:19.329 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:19.329 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:19.329 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:19.339 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:19.389 INFO:tasks.rados.rados.0.smithi204.stdout:1427: finishing write tid 2 to smithi20438004-18 2024-09-13T11:33:19.389 INFO:tasks.rados.rados.0.smithi204.stdout:1436: seq_num 286 ranges {406802=626275,1686203=728431} 2024-09-13T11:33:19.395 INFO:tasks.rados.rados.0.smithi204.stdout:1436: writing smithi20438004-5 from 406802 to 1033077 tid 1 2024-09-13T11:33:19.404 INFO:tasks.rados.rados.0.smithi204.stdout:1436: writing smithi20438004-5 from 1686203 to 2414634 tid 2 2024-09-13T11:33:19.404 INFO:tasks.rados.rados.0.smithi204.stdout:1426: done (10 left) 2024-09-13T11:33:19.404 INFO:tasks.rados.rados.0.smithi204.stdout:1428: done (9 left) 2024-09-13T11:33:19.404 INFO:tasks.rados.rados.0.smithi204.stdout:1429: done (8 left) 2024-09-13T11:33:19.404 INFO:tasks.rados.rados.0.smithi204.stdout:1437: write oid 32 current snap is 0 2024-09-13T11:33:19.404 INFO:tasks.rados.rados.0.smithi204.stdout:1427: finishing write tid 3 to smithi20438004-18 2024-09-13T11:33:19.404 INFO:tasks.rados.rados.0.smithi204.stdout:1427: finishing write tid 4 to smithi20438004-18 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1427: oid 18 updating version 0 to 109 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1427: oid 18 updating version 109 to 110 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1427: oid 18 updating version 110 to 111 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 111 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1427: left oid 18 (ObjNum 283 snap 0 seq_num 283) 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1430: finishing write tid 1 to smithi20438004-15 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1430: finishing write tid 2 to smithi20438004-15 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1430: finishing write tid 3 to smithi20438004-15 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1430: finishing write tid 4 to smithi20438004-15 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1430: oid 15 updating version 0 to 80 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1430: oid 15 updating version 80 to 81 2024-09-13T11:33:19.405 INFO:tasks.rados.rados.0.smithi204.stdout:1430: oid 15 updating version 81 to 82 2024-09-13T11:33:19.406 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 82 (ObjNum 284 snap 0 seq_num 284) dirty exists 2024-09-13T11:33:19.406 INFO:tasks.rados.rados.0.smithi204.stdout:1430: left oid 15 (ObjNum 284 snap 0 seq_num 284) 2024-09-13T11:33:19.406 INFO:tasks.rados.rados.0.smithi204.stdout:1437: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-13T11:33:19.409 INFO:tasks.rados.rados.0.smithi204.stdout:1437: writing smithi20438004-32 from 423609 to 1125133 tid 1 2024-09-13T11:33:19.414 INFO:tasks.rados.rados.0.smithi204.stdout:1437: writing smithi20438004-32 from 1628332 to 2100421 tid 2 2024-09-13T11:33:19.419 INFO:tasks.rados.rados.0.smithi204.stdout:1437: writing smithi20438004-32 from 2648987 to 2823609 tid 3 2024-09-13T11:33:19.419 INFO:tasks.rados.rados.0.smithi204.stdout:1427: done (8 left) 2024-09-13T11:33:19.419 INFO:tasks.rados.rados.0.smithi204.stdout:1430: done (7 left) 2024-09-13T11:33:19.419 INFO:tasks.rados.rados.0.smithi204.stdout:1438: read oid 7 snap -1 2024-09-13T11:33:19.420 INFO:tasks.rados.rados.0.smithi204.stdout:1438: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-13T11:33:19.420 INFO:tasks.rados.rados.0.smithi204.stdout:1439: read oid 44 snap -1 2024-09-13T11:33:19.420 INFO:tasks.rados.rados.0.smithi204.stdout:1439: expect deleted 2024-09-13T11:33:19.420 INFO:tasks.rados.rados.0.smithi204.stdout:1440: set_chunk oid 1 target oid 2024-09-13T11:33:19.424 INFO:tasks.rados.rados.0.smithi204.stdout:1440: set_chunk oid 1 offset: 169984 length: 282112 target oid 49cf32cf6411bc859f27f860eb52145789a9396f9683e6e8b42c211bdd067e59 offset: 0 snap -1 2024-09-13T11:33:19.425 INFO:tasks.rados.rados.0.smithi204.stdout:1441: tier_promote oid 2 2024-09-13T11:33:19.425 INFO:tasks.rados.rados.0.smithi204.stdout:1431: got (0) Success 2024-09-13T11:33:19.425 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 68 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-09-13T11:33:19.425 INFO:tasks.rados.rados.0.smithi204.stdout:1432: finishing set_chunk to oid 10 2024-09-13T11:33:19.425 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 114 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-13T11:33:19.425 INFO:tasks.rados.rados.0.smithi204.stdout:1433: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-13T11:33:19.426 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:19.425+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:19.431 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1202747 2024-09-13T11:33:19.449 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:19.449 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:19.450 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:19.450 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:19.450 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:19.459 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:19.478 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 552847 2024-09-13T11:33:19.527 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:19.527+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:19.567 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:19.567 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:19.567 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:19.568 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:19.568 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.568 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:19.568 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:19.568 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.210063324000004, 2024-09-13T11:33:19.568 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:33:19.569 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.569 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.569 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:19.569 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.569 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:19.570 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.571 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:33:19.572 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.573 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.574 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:33:19.575 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.576 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.577 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.577 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.577 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:19.577 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:19.577 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.461532403, 2024-09-13T11:33:19.577 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.578 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.579 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:33:19.580 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.581 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.582 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.449314809000001, 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:19.583 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.584 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.585 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:33:19.586 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.587 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.587 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.587 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:33:19.587 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:33:19.587 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.587 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.587 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.588 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.588 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.588 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1679 4.4 4:2b094039:::smithi20438004-40:head [tier-flush] snapc 0=[] RETRY=3 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:33:19.588 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:19.588 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.295190701999999, 2024-09-13T11:33:19.588 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0158098280000001, 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1679 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.589 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:19.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.590 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.590 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.590 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:19.590 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.590 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269187+0000", 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9479999999999998e-06 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269193+0000", 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.173e-06 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.591 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269209+0000", 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6039e-05 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269224+0000", 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5494000000000001e-05 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270767+0000", 2024-09-13T11:33:19.592 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015428869999999999 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270782+0000", 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.519e-05 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284573+0000", 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013791077000000001 2024-09-13T11:33:19.593 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284717+0000", 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014369599999999999 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284994+0000", 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.000277324 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.594 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2310 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.464569157, 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17044168200000001, 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:19.595 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2310 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099808+0000", 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100126+0000", 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031826000000000001 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100144+0000", 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8029999999999998e-05 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100153+0000", 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0119999999999994e-06 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100230+0000", 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6221000000000001e-05 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100242+0000", 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.257e-05 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270222+0000", 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16997934100000001 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270248+0000", 2024-09-13T11:33:19.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6366000000000001e-05 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2311 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.458028881000001, 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16403588, 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.601 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2311 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106347+0000", 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106673+0000", 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588799999999998 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106687+0000", 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3247e-05 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106693+0000", 2024-09-13T11:33:19.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2249999999999997e-06 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106741+0000", 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8715999999999999e-05 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106750+0000", 2024-09-13T11:33:19.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3159999999999997e-06 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270375+0000", 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16362501300000001 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270382+0000", 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.606 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2312 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.452737619000001, 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158778577, 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.607 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2312 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:19.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111639+0000", 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111756+0000", 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117166 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111770+0000", 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3492999999999999e-05 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111778+0000", 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5820000000000004e-06 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111834+0000", 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6564000000000003e-05 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111842+0000", 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8280000000000003e-06 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270410+0000", 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15856827400000001 2024-09-13T11:33:19.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270416+0000", 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2313 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.452583288, 2024-09-13T11:33:19.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158679564, 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2313 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6399999999999998e-07 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111795+0000", 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0450000000000001e-06 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111799+0000", 2024-09-13T11:33:19.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5250000000000001e-06 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111802+0000", 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.162e-06 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111876+0000", 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4062999999999999e-05 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.616 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111885+0000", 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0850000000000003e-06 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270465+0000", 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158579839 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270472+0000", 2024-09-13T11:33:19.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4810000000000003e-06 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4.8a282f38 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.452553072000001, 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15868154300000001, 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.618 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111823+0000", 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3299999999999998e-07 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111824+0000", 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7290000000000001e-06 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111829+0000", 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5859999999999998e-06 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111832+0000", 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.456e-06 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111901+0000", 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8447000000000004e-05 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111905+0000", 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8569999999999997e-06 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.622 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270497+0000", 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158592381 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270504+0000", 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.754e-06 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2309 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:33:19.623 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.295057954000001, 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.00703426, 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2309 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269319+0000", 2024-09-13T11:33:19.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.265e-06 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269324+0000", 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1800000000000004e-06 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269334+0000", 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7310000000000005e-06 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269343+0000", 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8339999999999994e-06 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269381+0000", 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7709000000000001e-05 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269391+0000", 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278002+0000", 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086114859999999998 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278150+0000", 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014796199999999999 2024-09-13T11:33:19.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.276352+0000", 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99820153300000003 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2170 4.12 4:4b088bf8:::smithi20438004-24:head [write 2083958~603055] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:19.629 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.286492092, 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029083859, 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2170 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277884+0000", 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4399999999999998e-07 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278124+0000", 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000239485 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278133+0000", 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3219999999999999e-06 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278141+0000", 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280144+0000", 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020033289999999999 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280155+0000", 2024-09-13T11:33:19.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0966e-05 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301300+0000", 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021145557999999998 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301414+0000", 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011306099999999999 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301601+0000", 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000187357 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304518+0000", 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029169539999999998 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304561+0000", 2024-09-13T11:33:19.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229000000000002e-05 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306715+0000", 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002153706 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306792+0000", 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7466999999999995e-05 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306851+0000", 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8811999999999999e-05 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306967+0000", 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116062 2024-09-13T11:33:19.637 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.640 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.640 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2171 4.12 4:4b088bf8:::smithi20438004-24:head [setxattr _header (56) in=7b,truncate 3058874] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.286219524, 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029067939000000001, 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2171 2024-09-13T11:33:19.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.642 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278157+0000", 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0699999999999996e-07 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278161+0000", 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2640000000000002e-06 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278173+0000", 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1868999999999999e-05 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278179+0000", 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280537+0000", 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023576320000000001 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280544+0000", 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7560000000000002e-06 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303651+0000", 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023107240000000001 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303802+0000", 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150774 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303982+0000", 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018009800000000001 2024-09-13T11:33:19.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306699+0000", 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027165100000000001 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306720+0000", 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1535e-05 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306753+0000", 2024-09-13T11:33:19.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3019999999999999e-05 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307010+0000", 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025724800000000001 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307061+0000", 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0633000000000001e-05 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307224+0000", 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000162843 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.648 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2172 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.286186839999999, 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029398245999999999, 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2172 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000005e-07 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278193+0000", 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278201+0000", 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278207+0000", 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4289999999999998e-06 2024-09-13T11:33:19.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280568+0000", 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023610900000000002 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280577+0000", 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8259999999999999e-06 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304605+0000", 2024-09-13T11:33:19.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024027829000000001 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304704+0000", 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9610000000000003e-05 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307334+0000", 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002629667 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307421+0000", 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7433000000000001e-05 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307587+0000", 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016528800000000001 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2219 4.12 4:4d4c8512:::smithi20438004-29:head [sparse-read 0~2348017] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:19.654 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.286115632000001, 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029874108999999999, 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2219 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3599999999999999e-07 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278262+0000", 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2110000000000001e-06 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278269+0000", 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.314e-06 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278274+0000", 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5249999999999996e-06 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280850+0000", 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025761389999999999 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280858+0000", 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8760000000000005e-06 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307628+0000", 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.026769465999999999 2024-09-13T11:33:19.658 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307703+0000", 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4847999999999999e-05 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.308134+0000", 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00043129399999999998 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.659 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.284360164999999, 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029229465, 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.660 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280016+0000", 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280020+0000", 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280026+0000", 2024-09-13T11:33:19.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1070000000000001e-06 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280032+0000", 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280109+0000", 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6298999999999996e-05 2024-09-13T11:33:19.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280116+0000", 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5129999999999999e-06 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298271+0000", 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018154805 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:19.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298334+0000", 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3268999999999993e-05 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309030+0000", 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010695566 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309089+0000", 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309245+0000", 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000155317 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2340 4.18 4:1d367c64:::smithi20438004-45:head [write 578207~550227] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.284332417999998, 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030475704999999999, 2024-09-13T11:33:19.666 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2340 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0399999999999996e-07 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280452+0000", 2024-09-13T11:33:19.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040812800000000001 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280461+0000", 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3829999999999998e-06 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280468+0000", 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0400000000000004e-06 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280506+0000", 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8037000000000002e-05 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280514+0000", 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.011e-06 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298673+0000", 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018159411 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298829+0000", 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015594 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.299023+0000", 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193838 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307549+0000", 2024-09-13T11:33:19.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085256289999999998 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307635+0000", 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6596999999999998e-05 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307689+0000", 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4027999999999998e-05 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307748+0000", 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8733999999999997e-05 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310255+0000", 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025070940000000001 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310375+0000", 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119691 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.673 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310519+0000", 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014364 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2341 4.18 4:1d367c64:::smithi20438004-45:head [write 1762410~779734] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.283890051, 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031687067999999999, 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.674 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2341 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280486+0000", 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281018+0000", 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053231899999999996 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281027+0000", 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.704e-06 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281034+0000", 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0659999999999998e-06 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281159+0000", 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124661 2024-09-13T11:33:19.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281168+0000", 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000001e-06 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302725+0000", 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021557478000000001 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302889+0000", 2024-09-13T11:33:19.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016421299999999999 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303087+0000", 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019817200000000001 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307806+0000", 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004718168 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307868+0000", 2024-09-13T11:33:19.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2775999999999997e-05 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307920+0000", 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1749000000000002e-05 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307971+0000", 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1343e-05 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311829+0000", 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038571769999999998 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312043+0000", 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021412199999999999 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312172+0000", 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012966900000000001 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.681 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2342 4.18 4:1d367c64:::smithi20438004-45:head [write 3286845~91362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.283322900000002, 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031505618999999999, 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.682 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2342 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281053+0000", 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2799999999999995e-07 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281104+0000", 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0550999999999998e-05 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281112+0000", 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0129999999999993e-06 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281118+0000", 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281191+0000", 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2738999999999996e-05 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281197+0000", 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309351+0000", 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028153293999999999 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309441+0000", 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0048000000000002e-05 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309594+0000", 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153609 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311315+0000", 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017209829999999999 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311795+0000", 2024-09-13T11:33:19.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048009300000000001 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312281+0000", 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048612699999999999 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312394+0000", 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112935 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312404+0000", 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0112000000000001e-05 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312449+0000", 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4650000000000001e-05 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312558+0000", 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108959 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2343 4.18 4:1d367c64:::smithi20438004-45:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.28324735, 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031723945000000003, 2024-09-13T11:33:19.690 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2343 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281129+0000", 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281132+0000", 2024-09-13T11:33:19.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.988e-06 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281137+0000", 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.95e-06 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281142+0000", 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4e-06 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281219+0000", 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6946000000000005e-05 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281225+0000", 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.601e-06 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310552+0000", 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029326497 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310643+0000", 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1416999999999996e-05 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310817+0000", 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000173463 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312229+0000", 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001412424 2024-09-13T11:33:19.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312324+0000", 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4264000000000004e-05 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312578+0000", 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025401699999999999 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312605+0000", 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7557000000000001e-05 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312650+0000", 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4496000000000002e-05 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312683+0000", 2024-09-13T11:33:19.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2898000000000002e-05 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312852+0000", 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000169436 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2344 4.18 4:1d367c64:::smithi20438004-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:19.698 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.283227603, 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032021714999999999, 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2344 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:19.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.15e-07 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281150+0000", 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.857e-06 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281155+0000", 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0579999999999997e-06 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:19.701 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281160+0000", 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0409999999999997e-06 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281248+0000", 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7979000000000006e-05 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281255+0000", 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5089999999999999e-06 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311169+0000", 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02991394 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311266+0000", 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7007000000000001e-05 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312894+0000", 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016280489999999999 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312976+0000", 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2052999999999995e-05 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.313169+0000", 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193807 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:19.704 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:19.706 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:19.705+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:19.707 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1434: finishing write tid 1 to smithi20438004-22 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1435: got (22) Invalid argument 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1434: finishing write tid 2 to smithi20438004-22 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1431: done (10 left) 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1432: done (9 left) 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1433: done (8 left) 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1435: done (7 left) 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1442: tier_promote oid 34 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1434: finishing write tid 3 to smithi20438004-22 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1434: finishing write tid 4 to smithi20438004-22 2024-09-13T11:33:19.710 INFO:tasks.rados.rados.0.smithi204.stdout:1434: oid 22 updating version 0 to 85 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1434: oid 22 version 85 is already newer than 84 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1434: oid 22 version 85 is already newer than 83 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 85 (ObjNum 285 snap 0 seq_num 285) dirty exists 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1434: left oid 22 (ObjNum 285 snap 0 seq_num 285) 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1436: finishing write tid 1 to smithi20438004-5 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1436: finishing write tid 2 to smithi20438004-5 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1436: finishing write tid 3 to smithi20438004-5 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1434: done (7 left) 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1443: read oid 49 snap -1 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1443: expect deleted 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1444: read oid 15 snap -1 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1444: expect (ObjNum 284 snap 0 seq_num 284) 2024-09-13T11:33:19.711 INFO:tasks.rados.rados.0.smithi204.stdout:1445: tier_promote oid 26 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1436: finishing write tid 4 to smithi20438004-5 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1436: oid 5 updating version 0 to 73 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1436: oid 5 version 73 is already newer than 72 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1436: oid 5 updating version 73 to 74 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 74 (ObjNum 286 snap 0 seq_num 286) dirty exists 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1436: left oid 5 (ObjNum 286 snap 0 seq_num 286) 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1437: finishing write tid 1 to smithi20438004-32 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1436: done (9 left) 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1446: read oid 50 snap -1 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1446: expect deleted 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1447: read oid 13 snap -1 2024-09-13T11:33:19.712 INFO:tasks.rados.rados.0.smithi204.stdout:1447: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1448: read oid 43 snap -1 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1448: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1449: read oid 9 snap -1 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1449: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1450: write oid 28 current snap is 0 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1450: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1450: writing smithi20438004-28 from 440416 to 1217189 tid 1 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1450: writing smithi20438004-28 from 1686814 to 2418914 tid 2 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1451: write oid 17 current snap is 0 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1451: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1451: writing smithi20438004-17 from 457223 to 1025598 tid 1 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1451: writing smithi20438004-17 from 1461649 to 1937407 tid 2 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1451: writing smithi20438004-17 from 2420186 to 2857223 tid 3 2024-09-13T11:33:19.713 INFO:tasks.rados.rados.0.smithi204.stdout:1452: set_chunk oid 10 target oid 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1452: set_chunk oid 10 offset: 361984 length: 446464 target oid 44365167af255a0f39440f9caf96c307d32c5892cd3a6fe1c8f8a042510a9407 offset: 0 snap -1 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: finishing write tid 2 to smithi20438004-32 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: finishing write tid 3 to smithi20438004-32 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1439: done (15 left) 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1453: write oid 12 current snap is 0 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: finishing write tid 4 to smithi20438004-32 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: finishing write tid 5 to smithi20438004-32 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: oid 32 updating version 0 to 60 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: oid 32 version 60 is already newer than 59 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: oid 32 updating version 60 to 62 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:1437: oid 32 version 62 is already newer than 61 2024-09-13T11:33:19.714 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 62 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-09-13T11:33:19.715 INFO:tasks.rados.rados.0.smithi204.stdout:1437: left oid 32 (ObjNum 287 snap 0 seq_num 287) 2024-09-13T11:33:19.715 INFO:tasks.rados.rados.0.smithi204.stdout:1438: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-13T11:33:19.715 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1035333 2024-09-13T11:33:19.723 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 947053 2024-09-13T11:33:19.752 INFO:tasks.rados.rados.0.smithi204.stdout:1440: finishing set_chunk to oid 1 2024-09-13T11:33:19.752 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 66 (ObjNum 241 snap 0 seq_num 241) dirty exists 2024-09-13T11:33:19.752 INFO:tasks.rados.rados.0.smithi204.stdout:1441: got (0) Success 2024-09-13T11:33:19.752 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 86 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-09-13T11:33:19.752 INFO:tasks.rados.rados.0.smithi204.stdout:1453: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-13T11:33:19.759 INFO:tasks.rados.rados.0.smithi204.stdout:1453: writing smithi20438004-12 from 474030 to 1117654 tid 1 2024-09-13T11:33:19.768 INFO:tasks.rados.rados.0.smithi204.stdout:1453: writing smithi20438004-12 from 1520131 to 2255900 tid 2 2024-09-13T11:33:19.768 INFO:tasks.rados.rados.0.smithi204.stdout:1437: done (15 left) 2024-09-13T11:33:19.768 INFO:tasks.rados.rados.0.smithi204.stdout:1438: done (14 left) 2024-09-13T11:33:19.768 INFO:tasks.rados.rados.0.smithi204.stdout:1440: done (13 left) 2024-09-13T11:33:19.768 INFO:tasks.rados.rados.0.smithi204.stdout:1441: done (12 left) 2024-09-13T11:33:19.768 INFO:tasks.rados.rados.0.smithi204.stdout:1454: read oid 47 snap -1 2024-09-13T11:33:19.768 INFO:tasks.rados.rados.0.smithi204.stdout:1454: expect deleted 2024-09-13T11:33:19.769 INFO:tasks.rados.rados.0.smithi204.stdout:1455: read oid 5 snap -1 2024-09-13T11:33:19.769 INFO:tasks.rados.rados.0.smithi204.stdout:1455: expect (ObjNum 286 snap 0 seq_num 286) 2024-09-13T11:33:19.769 INFO:tasks.rados.rados.0.smithi204.stdout:1456: read oid 35 snap -1 2024-09-13T11:33:19.769 INFO:tasks.rados.rados.0.smithi204.stdout:1456: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:19.769 INFO:tasks.rados.rados.0.smithi204.stdout:1457: tier_promote oid 18 2024-09-13T11:33:19.769 INFO:tasks.rados.rados.0.smithi204.stdout:1448: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:19.769 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:19.806 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:19.805+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:19.834 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:19.834 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:19.834 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:19.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:19.861 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1569244 2024-09-13T11:33:19.907 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:19.906+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:19.963 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:19.963 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:19.963 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:19.963 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:19.963 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:19.971 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:19.987 INFO:tasks.rados.rados.0.smithi204.stdout:1449: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-13T11:33:20.007 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:20.005+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:20.024 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2175515 2024-09-13T11:33:20.076 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:20.076 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:20.076 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:20.076 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:20.076 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.977027640000003, 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.077 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.078 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:33:20.079 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:33:20.080 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.081 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.964593495000003, 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.082 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.083 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:33:20.084 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.085 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:20.086 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.954639901, 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.087 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:33:20.088 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:33:20.089 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.090 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.923922093999998, 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.091 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.092 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.092 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.092 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.092 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:20.092 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.093 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.093 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.093 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:33:20.094 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:33:20.095 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:33:20.096 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.097 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.905098774999999, 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:20.098 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.099 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.100 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.101 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.891579426, 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:33:20.102 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.103 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:33:20.104 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.105 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.106 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.881939093, 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:20.107 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.108 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.109 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:33:20.110 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.871633592999999, 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:33:20.111 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.112 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:33:20.113 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:33:20.114 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.115 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.861247650999999, 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.116 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:33:20.117 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.118 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.119 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:36 3.3 3:e2816a99:::d4bb749789c90c884ffd20ac62f804c7a6b80502d14432fa627e1239249ee2da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.803720251999998, 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013305613000000001, 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.120 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.121 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270480+0000", 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2399999999999998e-07 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270655+0000", 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017481300000000001 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270683+0000", 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8340999999999999e-05 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.122 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270693+0000", 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271029+0000", 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033640200000000002 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271038+0000", 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.123 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278355+0000", 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073166510000000004 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278481+0000", 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125955 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278657+0000", 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176816 2024-09-13T11:33:20.124 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280615+0000", 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019576810000000002 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283538+0000", 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029229030000000001 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.125 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283592+0000", 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4366000000000001e-05 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283671+0000", 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8337000000000006e-05 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283785+0000", 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114412 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.126 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:40 3.3 3:c7a12847:::768a08ae37e9bf19369a517906ebd209c1f2b8ad3684596d644da7d48b42114d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.803477145999999, 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013347114, 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.127 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:20.128 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270839+0000", 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115369 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270847+0000", 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270854+0000", 2024-09-13T11:33:20.129 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271073+0000", 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021964400000000001 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271080+0000", 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7909999999999999e-06 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278928+0000", 2024-09-13T11:33:20.130 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078479819999999999 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279058+0000", 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130102 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279300+0000", 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024166300000000001 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280641+0000", 2024-09-13T11:33:20.131 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013409229999999999 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283545+0000", 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002904401 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283927+0000", 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038193199999999997 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.132 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283964+0000", 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7117999999999999e-05 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284070+0000", 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010545899999999999 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.133 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:41 3.3 3:e9fa9a15:::3c65c6c9770b7a29cd5a9b6dc5b0b39ab23cb2a64bfb19442efcb911d255e067:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.803333533, 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013499547000000001, 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.134 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270867+0000", 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270893+0000", 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6057000000000001e-05 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.135 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270901+0000", 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270915+0000", 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4450000000000001e-05 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271104+0000", 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018904600000000001 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.136 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271111+0000", 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6039999999999998e-06 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280674+0000", 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095629140000000005 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280811+0000", 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136802 2024-09-13T11:33:20.137 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281027+0000", 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000216617 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283802+0000", 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027746540000000001 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283884+0000", 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2167000000000001e-05 2024-09-13T11:33:20.138 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284178+0000", 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029347899999999997 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284246+0000", 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8609999999999995e-05 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284366+0000", 2024-09-13T11:33:20.139 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011950799999999999 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:43 3.3 3:dc8816b1:::1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.803272837999998, 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013665096999999999, 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.140 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:20.141 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270933+0000", 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2410000000000003e-06 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270939+0000", 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5810000000000002e-06 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.142 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270945+0000", 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271134+0000", 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000189161 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271140+0000", 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2999999999999998e-06 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.143 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281407+0000", 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010266226 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281549+0000", 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142583 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281737+0000", 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018791500000000001 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.144 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283810+0000", 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002072571 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284127+0000", 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000317516 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284417+0000", 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029000899999999998 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.145 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284480+0000", 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3136e-05 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284592+0000", 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111712 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.146 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.10.22:39 3.3 3:c55f8587:::a2f2c330500f1b1d28c41e464817565848095218a89d32db0b731691bbe2b77b:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.781077068999998, 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011011337, 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.147 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293123+0000", 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293124+0000", 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5319999999999999e-06 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293132+0000", 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.148 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293143+0000", 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1477999999999999e-05 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.293152+0000", 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8289999999999997e-06 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298197+0000", 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050456870000000001 2024-09-13T11:33:20.149 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298273+0000", 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5799999999999999e-05 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.298497+0000", 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000223618 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:20.150 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.299173+0000", 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00067642199999999998 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.299376+0000", 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020231700000000001 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304021+0000", 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004645003 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.151 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304068+0000", 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7139999999999999e-05 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.304134+0000", 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6203000000000003e-05 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:20.152 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 24.229444965999999, 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43316489499999999, 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:20.153 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844756+0000", 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.154 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844760+0000", 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0550000000000001e-06 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844769+0000", 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6129999999999993e-06 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844776+0000", 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0280000000000003e-06 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.155 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7639000000000002e-05 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844841+0000", 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1049999999999997e-06 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277896+0000", 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43305493499999997 2024-09-13T11:33:20.156 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277920+0000", 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3614000000000001e-05 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 24.229393333000001, 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43315226600000001, 2024-09-13T11:33:20.157 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.158 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844807+0000", 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3200000000000003e-07 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844811+0000", 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.799e-06 2024-09-13T11:33:20.159 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844827+0000", 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5461999999999999e-05 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.126e-06 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.160 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5369999999999999e-05 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844863+0000", 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.658e-06 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.161 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277951+0000", 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308847900000003 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277959+0000", 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:20.162 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 24.229351621999999, 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43314228399999999, 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.163 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844849+0000", 2024-09-13T11:33:20.164 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0100000000000005e-07 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844853+0000", 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9589999999999997e-06 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4110000000000001e-06 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.165 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844864+0000", 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8930000000000004e-06 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844892+0000", 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8042e-05 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844898+0000", 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5049999999999998e-06 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.166 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277984+0000", 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308647700000003 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277990+0000", 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3960000000000003e-06 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4:8c30e60e:::smithi20438004-18:head [read 0~2487436,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:20.167 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.791802300000001, 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012791258999999999, 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.168 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282399+0000", 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.691e-06 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.169 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282404+0000", 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.887e-06 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282414+0000", 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9539999999999999e-06 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282423+0000", 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6039999999999993e-06 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.170 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282477+0000", 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4738999999999999e-05 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282486+0000", 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5450000000000003e-06 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287493+0000", 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050068949999999999 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.171 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287636+0000", 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014333299999999999 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.295189+0000", 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075526109999999999 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.172 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 153896~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.791700710000001, 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014013433, 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.173 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282500+0000", 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5499999999999998e-07 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.174 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282503+0000", 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.563e-06 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282510+0000", 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8820000000000003e-06 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282516+0000", 2024-09-13T11:33:20.175 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282562+0000", 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5868999999999997e-05 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282570+0000", 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296085+0000", 2024-09-13T11:33:20.176 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013515255 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296187+0000", 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101457 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296512+0000", 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588099999999998 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.177 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 447518~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 22.791616217000001, 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014496826000000001, 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:20.178 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.179 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5000000000000002e-07 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282589+0000", 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4220000000000002e-06 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282595+0000", 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8789999999999997e-06 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.180 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282606+0000", 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0229e-05 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282652+0000", 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6610999999999998e-05 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282661+0000", 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2640000000000008e-06 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.181 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296550+0000", 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013888981999999999 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296628+0000", 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8564000000000002e-05 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.297080+0000", 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000452225 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.182 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.183 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.183 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.183 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.183 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:20.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:20.185 INFO:tasks.rados.rados.0.smithi204.stdout:1442: got (0) Success 2024-09-13T11:33:20.185 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 103 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-13T11:33:20.185 INFO:tasks.rados.rados.0.smithi204.stdout:1447: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:20.185 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 595523 2024-09-13T11:33:20.186 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:20.183+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:20.285 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:20.284+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:20.297 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:33:20.316 INFO:tasks.rados.rados.0.smithi204.stdout:1445: got (0) Success 2024-09-13T11:33:20.316 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 56 (ObjNum 248 snap 0 seq_num 248) dirty exists 2024-09-13T11:33:20.316 INFO:tasks.rados.rados.0.smithi204.stdout:1444: expect (ObjNum 284 snap 0 seq_num 284) 2024-09-13T11:33:20.316 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 437785 2024-09-13T11:33:20.325 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:20.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:20.325 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:20.326 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T11:33:20.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:33:20.329 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:20.329 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:20.329 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:20.337 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:20.386 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:20.384+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:20.388 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2099776 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: finishing write tid 1 to smithi20438004-28 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: finishing write tid 2 to smithi20438004-28 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: finishing write tid 3 to smithi20438004-28 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: finishing write tid 4 to smithi20438004-28 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: oid 28 updating version 0 to 77 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: oid 28 version 77 is already newer than 76 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: oid 28 version 77 is already newer than 75 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 77 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1450: left oid 28 (ObjNum 288 snap 0 seq_num 288) 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1451: finishing write tid 1 to smithi20438004-17 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1442: done (15 left) 2024-09-13T11:33:20.397 INFO:tasks.rados.rados.0.smithi204.stdout:1443: done (14 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1444: done (13 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1445: done (12 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1446: done (11 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1447: done (10 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1448: done (9 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1449: done (8 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1450: done (7 left) 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1458: tier_evict oid 50 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1458: tier_evict oid 50 snap -1 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1451: finishing write tid 2 to smithi20438004-17 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1451: finishing write tid 3 to smithi20438004-17 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1451: finishing write tid 4 to smithi20438004-17 2024-09-13T11:33:20.398 INFO:tasks.rados.rados.0.smithi204.stdout:1452: finishing set_chunk to oid 10 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 361984 length: 446464 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1451: finishing write tid 5 to smithi20438004-17 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1451: oid 17 updating version 0 to 78 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1451: oid 17 updating version 78 to 80 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1451: oid 17 version 80 is already newer than 77 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1451: oid 17 version 80 is already newer than 79 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 80 (ObjNum 289 snap 0 seq_num 289) dirty exists 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1451: left oid 17 (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1453: finishing write tid 1 to smithi20438004-12 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1453: finishing write tid 2 to smithi20438004-12 2024-09-13T11:33:20.399 INFO:tasks.rados.rados.0.smithi204.stdout:1455: expect (ObjNum 286 snap 0 seq_num 286) 2024-09-13T11:33:20.428 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 239104 2024-09-13T11:33:20.441 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:20.442 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:20.442 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:20.442 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:20.442 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:20.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:20.486 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:20.485+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:20.502 INFO:tasks.rados.rados.0.smithi204.stdout:1456: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:20.588 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:20.587+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:20.594 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.491759893000001, 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016706373, 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.595 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.596 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.99e-07 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104537+0000", 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034575700000000001 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104552+0000", 2024-09-13T11:33:20.597 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5475999999999999e-05 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104566+0000", 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3766e-05 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104623+0000", 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7046999999999998e-05 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.598 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104650+0000", 2024-09-13T11:33:20.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6786000000000001e-05 2024-09-13T11:33:20.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.599 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.599 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.599 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117752+0000", 2024-09-13T11:33:20.599 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013102387 2024-09-13T11:33:20.599 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117785+0000", 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2741999999999999e-05 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.600 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.487800567000001, 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025912691000000002, 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.601 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.602 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:20.602 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.602 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.602 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.602 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.603 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105039+0000", 2024-09-13T11:33:20.603 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7100000000000001e-07 2024-09-13T11:33:20.603 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108050+0000", 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030115910000000001 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108061+0000", 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0512e-05 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.604 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108069+0000", 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4479999999999994e-06 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108126+0000", 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7024999999999997e-05 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.605 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108165+0000", 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8701000000000001e-05 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130916+0000", 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022750829 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130951+0000", 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4913999999999998e-05 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.606 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:203 3.1 e38/30 3:970ba768:::smithi20438004-4 low_tier:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.484594074, 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031827052000000002, 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:20.607 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108245+0000", 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1800000000000005e-07 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.608 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111202+0000", 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029571440000000001 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111221+0000", 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8887000000000001e-05 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.609 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111248+0000", 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6458e-05 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111707+0000", 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045935799999999999 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111729+0000", 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.610 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140022+0000", 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028292571999999998 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140071+0000", 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9669000000000002e-05 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:20.611 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.481570523999999, 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028930938999999999, 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.612 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111269+0000", 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5700000000000002e-07 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113854+0000", 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025854340000000002 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.613 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113866+0000", 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1884e-05 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113873+0000", 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116175+0000", 2024-09-13T11:33:20.614 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023019 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116218+0000", 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2796000000000002e-05 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140165+0000", 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023947718999999999 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140199+0000", 2024-09-13T11:33:20.615 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3596000000000002e-05 2024-09-13T11:33:20.616 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.616 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.616 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.616 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.616 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.616 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:218 3.1 e38/30 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.471707616000003, 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025863384999999999, 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.617 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121132+0000", 2024-09-13T11:33:20.618 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3260000000000001e-06 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124734+0000", 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036019379999999998 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.619 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124748+0000", 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913999999999999e-05 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124760+0000", 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1786e-05 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124849+0000", 2024-09-13T11:33:20.620 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8882999999999996e-05 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124874+0000", 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5466999999999999e-05 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146945+0000", 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022070837999999999 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.621 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146994+0000", 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9233000000000001e-05 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.464774433000002, 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020872166000000001, 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.622 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128066+0000", 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.86e-06 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.623 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131456+0000", 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033901360000000002 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131467+0000", 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0806e-05 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131474+0000", 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-13T11:33:20.624 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131524+0000", 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9262999999999999e-05 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131554+0000", 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0148999999999999e-05 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148883+0000", 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017329445999999998 2024-09-13T11:33:20.625 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148936+0000", 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2879e-05 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.461336234000001, 2024-09-13T11:33:20.626 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017580887, 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.627 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131503+0000", 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133995+0000", 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024920630000000001 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134006+0000", 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0696000000000001e-05 2024-09-13T11:33:20.628 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134012+0000", 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136541+0000", 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025286150000000001 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136563+0000", 2024-09-13T11:33:20.629 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2531000000000002e-05 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149031+0000", 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012467951 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.630 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149083+0000", 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2040999999999999e-05 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:20.631 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.435127655999999, 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022274233000000001, 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.632 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157712+0000", 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3689999999999999e-06 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161974+0000", 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042619160000000001 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.633 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161990+0000", 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5506e-05 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161998+0000", 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7520000000000002e-06 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.634 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162103+0000", 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104415 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162126+0000", 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3643e-05 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179932+0000", 2024-09-13T11:33:20.635 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017805476000000001 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179985+0000", 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3155999999999997e-05 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:20.636 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.430802423000003, 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018684003000000001, 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.637 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162037+0000", 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.083e-06 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.638 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166161+0000", 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041238799999999999 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166179+0000", 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8329999999999999e-05 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166190+0000", 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0822000000000001e-05 2024-09-13T11:33:20.639 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166302+0000", 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011233700000000001 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166328+0000", 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5185999999999999e-05 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.640 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180646+0000", 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014318261000000001 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180720+0000", 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.641 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.426623421000002, 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014657148, 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.642 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1300000000000001e-07 2024-09-13T11:33:20.643 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169754+0000", 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035389779999999999 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169772+0000", 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8020999999999999e-05 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.644 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0037999999999999e-05 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169847+0000", 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4695000000000003e-05 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169865+0000", 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7365999999999999e-05 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.645 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180834+0000", 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010969016 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180872+0000", 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8621000000000002e-05 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.646 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.414712268000002, 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018123856000000001, 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.647 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.648 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:20.648 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.648 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.648 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.648 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178128+0000", 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181360+0000", 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032324699999999999 2024-09-13T11:33:20.649 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181371+0000", 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0978000000000001e-05 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181378+0000", 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.861e-06 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.650 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181431+0000", 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3173000000000001e-05 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181455+0000", 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4236999999999999e-05 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196213+0000", 2024-09-13T11:33:20.651 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01475807 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196250+0000", 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6636e-05 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:20.652 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.411428710999999, 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016397140000000001, 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:20.653 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181411+0000", 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7399999999999991e-07 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184468+0000", 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030575759999999998 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.654 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184481+0000", 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3072e-05 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184491+0000", 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184550+0000", 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8251999999999997e-05 2024-09-13T11:33:20.655 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184575+0000", 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689000000000001e-05 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197720+0000", 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013144708999999999 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.656 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197807+0000", 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6799999999999996e-05 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:281 4.a 4:56ef4fd4:::smithi20438004-32:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.360389998999999, 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018358335999999999, 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.657 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.658 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:20.658 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.660 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.660 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.661 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232449+0000", 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7499999999999998e-07 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232457+0000", 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3610000000000003e-06 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232470+0000", 2024-09-13T11:33:20.662 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3230999999999999e-05 2024-09-13T11:33:20.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.663 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.663 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.663 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232478+0000", 2024-09-13T11:33:20.663 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5989999999999996e-06 2024-09-13T11:33:20.663 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232541+0000", 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3717e-05 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232619+0000", 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:33:20.664 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248106+0000", 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015486616 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248991+0000", 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088509199999999998 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249363+0000", 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037176599999999999 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.665 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250719+0000", 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001356255 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250765+0000", 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6053000000000001e-05 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250807+0000", 2024-09-13T11:33:20.666 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1675000000000003e-05 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1696 4.b 4:da92e835:::smithi20438004-50:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.324238184000002, 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016782392, 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.667 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1696 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.668 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268602+0000", 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268608+0000", 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7629999999999997e-06 2024-09-13T11:33:20.669 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268620+0000", 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1950000000000001e-05 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268632+0000", 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2330999999999999e-05 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.670 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269210+0000", 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057761900000000003 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269221+0000", 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.168e-05 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277483+0000", 2024-09-13T11:33:20.671 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0082621180000000006 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277641+0000", 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000157365 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285383+0000", 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00774195 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.672 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1726 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.324169726999997, 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015213417999999999, 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1726 2024-09-13T11:33:20.673 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.674 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.58e-07 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268673+0000", 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7809999999999999e-06 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268679+0000", 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7729999999999996e-06 2024-09-13T11:33:20.675 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268684+0000", 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269275+0000", 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059124500000000001 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.676 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269281+0000", 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2779999999999999e-06 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281010+0000", 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011728762 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281114+0000", 2024-09-13T11:33:20.677 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103872 2024-09-13T11:33:20.678 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.678 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.678 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.678 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283882+0000", 2024-09-13T11:33:20.678 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027680539999999998 2024-09-13T11:33:20.678 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.678 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1727 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 34.324147979999999, 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016698689999999999, 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.679 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1727 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.680 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268691+0000", 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6100000000000001e-07 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268694+0000", 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.672e-06 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.681 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268699+0000", 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1819999999999997e-06 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268703+0000", 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269302+0000", 2024-09-13T11:33:20.682 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059838199999999995 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269307+0000", 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3170000000000001e-06 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284218+0000", 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014911308999999999 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284330+0000", 2024-09-13T11:33:20.683 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111715 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285389+0000", 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010592729999999999 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1775 4.a 4:532cb07c:::smithi20438004-26:head [sparse-read 1243799~869921] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273601+0000", 2024-09-13T11:33:20.684 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.319237018999999, 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012147214999999999, 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1775 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.685 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273601+0000", 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273601+0000", 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273602+0000", 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6700000000000003e-07 2024-09-13T11:33:20.686 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273606+0000", 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6480000000000001e-06 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273612+0000", 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6239999999999996e-06 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.687 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273618+0000", 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7710000000000003e-06 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273659+0000", 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0837000000000002e-05 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273666+0000", 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0620000000000003e-06 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.688 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285482+0000", 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011816491 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285559+0000", 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6854000000000007e-05 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.689 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.285749+0000", 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018936100000000001 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2018 4.5 4:a9b0c9bd:::smithi20438004-37:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.717180302999999, 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021048396, 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.690 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2018 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.691 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875659+0000", 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.692 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875661+0000", 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9920000000000002e-06 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875665+0000", 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.693 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875668+0000", 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875714+0000", 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5189999999999999e-05 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875770+0000", 2024-09-13T11:33:20.694 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6824999999999999e-05 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.896707+0000", 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020936156000000001 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2574 4.a 4:532cb07c:::smithi20438004-26:head [read 363008~2507776] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:20.695 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 18.418142753000001, 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013408788, 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2574 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.696 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174696+0000", 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9699999999999996e-07 2024-09-13T11:33:20.697 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174698+0000", 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0169999999999999e-06 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174703+0000", 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.989e-06 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.698 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.174708+0000", 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5329999999999996e-06 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.185700+0000", 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010992500000000001 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.185762+0000", 2024-09-13T11:33:20.699 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2199000000000005e-05 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:02.188104+0000", 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023420530000000002 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.700 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3273 4.a 4:532cb07c:::smithi20438004-26:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:19.584591+0000", 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 1.00824794, 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012076857999999999, 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3273 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.701 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584591+0000", 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584591+0000", 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.702 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584592+0000", 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.762e-06 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584601+0000", 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7539999999999995e-06 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584623+0000", 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1659e-05 2024-09-13T11:33:20.703 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584633+0000", 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0179e-05 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584701+0000", 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7755000000000004e-05 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:20.704 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.584811+0000", 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110524 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:19.596667+0000", 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011856225 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.705 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.706 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:20.706 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:20.706 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:20.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 75, 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "b332d022-c5e6-4682-9d3d-2b6994cd0f20", 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-09-13T11:31:50.491934+0000", 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-09-13T11:33:20.290015+0000", 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-09-13T11:32:56.264138+0000", 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-09-13T11:32:44.537150+0000", 2024-09-13T11:33:20.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5806080, 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "noscrub", 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 7, 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:33:20.711 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 4, 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 16, 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-09-13T11:33:20.712 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:02.947125+0000", 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:20.713 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:20.714 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "25", 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:20.715 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:20.716 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:20.717 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-09-13T11:33:20.718 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 15.789999961853027, 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 15.789999961853027, 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.18999999761581421, 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-09-13T11:33:20.719 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:05.120118+0000", 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:20.720 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-09-13T11:33:20.721 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "28", 2024-09-13T11:33:20.722 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 28, 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:20.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:20.724 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:20.725 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 3.9700000286102295, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 3.9700000286102295, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.5, 2024-09-13T11:33:20.726 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.5, 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "low_tier", 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:17.683275+0000", 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-09-13T11:33:20.727 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 4, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 4, 2024-09-13T11:33:20.728 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 4, 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 4, 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 4, 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "32", 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:20.729 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:20.730 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.731 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-09-13T11:33:20.732 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": {}, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:18.790550+0000", 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:20.734 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 26, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 26, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 26, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 26, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 26, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:20.735 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "58", 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "56", 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:20.736 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:20.737 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:20.738 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_cdc_chunk_size": 131072, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_tier": 3, 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: "fingerprint_algorithm": "sha256" 2024-09-13T11:33:20.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 3.6700000762939453, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 3.6700000762939453, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.309999942779541, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.309999942779541, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:20.740 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "5a4d0d43-6562-441e-925e-4c992ee7b335", 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.741 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:20.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.743 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.744 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2677506544", 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2677506544", 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2677506544", 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2677506544", 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "80665caa-6933-4f89-b454-128057e9ad92", 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.746 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 41, 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6808", 2024-09-13T11:33:20.747 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6809", 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6811", 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.749 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6810", 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6808/3561806747", 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6809/3561806747", 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6811/3561806747", 2024-09-13T11:33:20.750 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6810/3561806747", 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "eb99fd51-a064-4311-932e-8b6d555409af", 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.751 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 21, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 0, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 41, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.752 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6800", 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6801", 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.753 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6803", 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.754 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6802", 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6800/4217282489", 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6801/4217282489", 2024-09-13T11:33:20.755 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6803/4217282489", 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6802/4217282489", 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "3d840ed9-a410-4bcb-b0f0-70fafdf69842", 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.756 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.636688232421875, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.757 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6804", 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6805", 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.758 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6807", 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.759 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6806", 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6804/2326505959", 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6805/2326505959", 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2326505959", 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2326505959", 2024-09-13T11:33:20.760 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0c01b7d7-1309-40fa-913a-97d2a8499293", 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.761 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 72, 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.763 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:20.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/3107658120", 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/3107658120", 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/3107658120", 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/3107658120", 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "b4cec928-1764-4fe0-8c5d-870d8a0c15c7", 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 48, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.766 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6800", 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6801", 2024-09-13T11:33:20.767 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6803", 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.768 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6802", 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6800/3800782738", 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6801/3800782738", 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6803/3800782738", 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6802/3800782738", 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.769 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "89f6d810-2afc-42b0-b989-052218b6887c", 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6808", 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6809", 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6811", 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:20.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6810", 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6808/3986362716", 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6809/3986362716", 2024-09-13T11:33:20.773 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6811/3986362716", 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6810/3986362716", 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "ccfee49f-f394-4de0-8e30-853f38e31232", 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.774 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 23, 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6808", 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6809", 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.776 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6811", 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6810", 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.777 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6808/1978982202", 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6809/1978982202", 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6811/1978982202", 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6810/1978982202", 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 8, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "03b1c505-1b92-4daa-9da4-c254be2ba792", 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 41, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.779 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.780 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/3608171775", 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/3608171775", 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/3608171775", 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/3608171775", 2024-09-13T11:33:20.782 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 9, 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "8fbee7a0-2ee6-4551-bf06-4e013e413fb4", 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 48, 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6804", 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.784 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6805", 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6807", 2024-09-13T11:33:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6806", 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.786 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6804/2580659844", 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6805/2580659844", 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6807/2580659844", 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6806/2580659844", 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 10, 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0808cb41-f62e-49b0-935d-616bea1a4db5", 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.787 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.788 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6812", 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6813", 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6815", 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.790 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6814", 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6812/2352891695", 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6813/2352891695", 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6815/2352891695", 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6814/2352891695", 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 11, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "427dd574-f2a8-49e5-a32a-cfda023a514a", 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 21, 2024-09-13T11:33:20.792 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 72, 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6800", 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6801", 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6803", 2024-09-13T11:33:20.794 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6802", 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.795 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6800/2304836889", 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6801/2304836889", 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6803/2304836889", 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6802/2304836889", 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 12, 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "17942689-1130-4b6b-9f98-1605a03731b1", 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-09-13T11:33:20.796 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 58, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 59, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.797 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.799 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1688419254", 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1688419254", 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1688419254", 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1688419254", 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-09-13T11:33:20.800 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 13, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "62161cab-6d14-4571-971d-217601e38fbc", 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 23, 2024-09-13T11:33:20.801 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 48, 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6812", 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.802 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6813", 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6815", 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.803 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6814", 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6812/3727425991", 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6813/3727425991", 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6815/3727425991", 2024-09-13T11:33:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6814/3727425991", 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 14, 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "559ef141-27db-452d-9577-a6135421f64f", 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.805 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6804", 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6805", 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6807", 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.808 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6806", 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6804/538351706", 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6805/538351706", 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6807/538351706", 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6806/538351706", 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 15, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "98f9ceba-79df-410a-9788-393057e79c30", 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 23, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 72, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:20.810 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6812", 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6813", 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6815", 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.812 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6814", 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6812/501019151", 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6813/501019151", 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6815/501019151", 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6814/501019151", 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:20.813 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:03.855267+0000", 2024-09-13T11:33:20.814 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:00.915157+0000", 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-09-13T11:33:20.815 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:00.198442+0000", 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 41 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.816 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:04.310901+0000", 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.817 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:03.659118+0000", 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:09.623455+0000", 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.818 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.808086+0000", 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.819 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.499908+0000", 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 8, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:07.828658+0000", 2024-09-13T11:33:20.820 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 9, 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:00.655711+0000", 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.821 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 10, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:01.092457+0000", 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 11, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.822 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:00.246352+0000", 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 12, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:01.027609+0000", 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 59 2024-09-13T11:33:20.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 13, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.963999+0000", 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 14, 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.824 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.534738+0000", 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 15, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:02.472566+0000", 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [ 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "3.3", 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: 15, 2024-09-13T11:33:20.826 INFO:teuthology.orchestra.run.smithi038.stdout: 13 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.2", 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: 12, 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: 9 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.4", 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: 8, 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.11", 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-09-13T11:33:20.828 INFO:teuthology.orchestra.run.smithi038.stdout: 15 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.12", 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: 14, 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: 8, 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: 3 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.829 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.13", 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: 14, 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: 6 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.14", 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: 4, 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: 3 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.16", 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: 10, 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: 6, 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: 12 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.19", 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: 6, 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-09-13T11:33:20.831 INFO:teuthology.orchestra.run.smithi038.stdout: 7 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-09-13T11:33:20.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:20.833 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:20.835 INFO:tasks.thrashosds.thrasher:j is {'epoch': 75, 'fsid': 'b332d022-c5e6-4682-9d3d-2b6994cd0f20', 'created': '2024-09-13T11:31:50.491934+0000', 'modified': '2024-09-13T11:33:20.290015+0000', 'last_up_change': '2024-09-13T11:32:56.264138+0000', 'last_in_change': '2024-09-13T11:32:44.537150+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:32:02.947125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.789999961853027, 'score_stable': 15.789999961853027, 'optimal_score': 0.1899999976158142, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:32:05.120118+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-09-13T11:32:17.683275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:32:18.790550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '56', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.6700000762939453, 'score_stable': 3.6700000762939453, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.309999942779541, 'raw_score_stable': 2.309999942779541, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '5a4d0d43-6562-441e-925e-4c992ee7b335', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6805', 'nonce': 2677506544}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 2677506544}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 2677506544}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 2677506544}]}, 'public_addr': '172.21.15.38:6805/2677506544', 'cluster_addr': '172.21.15.38:6806/2677506544', 'heartbeat_back_addr': '172.21.15.38:6808/2677506544', 'heartbeat_front_addr': '172.21.15.38:6807/2677506544', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80665caa-6933-4f89-b454-128057e9ad92', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 3561806747}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6809', 'nonce': 3561806747}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6811', 'nonce': 3561806747}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 3561806747}]}, 'public_addr': '172.21.15.118:6808/3561806747', 'cluster_addr': '172.21.15.118:6809/3561806747', 'heartbeat_back_addr': '172.21.15.118:6811/3561806747', 'heartbeat_front_addr': '172.21.15.118:6810/3561806747', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'eb99fd51-a064-4311-932e-8b6d555409af', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 41, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 4217282489}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 4217282489}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 4217282489}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 4217282489}]}, 'public_addr': '172.21.15.154:6800/4217282489', 'cluster_addr': '172.21.15.154:6801/4217282489', 'heartbeat_back_addr': '172.21.15.154:6803/4217282489', 'heartbeat_front_addr': '172.21.15.154:6802/4217282489', 'state': ['exists']}, {'osd': 3, 'uuid': '3d840ed9-a410-4bcb-b0f0-70fafdf69842', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.636688232421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 2326505959}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 2326505959}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 2326505959}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 2326505959}]}, 'public_addr': '172.21.15.204:6804/2326505959', 'cluster_addr': '172.21.15.204:6805/2326505959', 'heartbeat_back_addr': '172.21.15.204:6807/2326505959', 'heartbeat_front_addr': '172.21.15.204:6806/2326505959', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0c01b7d7-1309-40fa-913a-97d2a8499293', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6801', 'nonce': 3107658120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 3107658120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6804', 'nonce': 3107658120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 3107658120}]}, 'public_addr': '172.21.15.38:6801/3107658120', 'cluster_addr': '172.21.15.38:6802/3107658120', 'heartbeat_back_addr': '172.21.15.38:6804/3107658120', 'heartbeat_front_addr': '172.21.15.38:6803/3107658120', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4cec928-1764-4fe0-8c5d-870d8a0c15c7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 3800782738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6801', 'nonce': 3800782738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6803', 'nonce': 3800782738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 3800782738}]}, 'public_addr': '172.21.15.118:6800/3800782738', 'cluster_addr': '172.21.15.118:6801/3800782738', 'heartbeat_back_addr': '172.21.15.118:6803/3800782738', 'heartbeat_front_addr': '172.21.15.118:6802/3800782738', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89f6d810-2afc-42b0-b989-052218b6887c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 3986362716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 3986362716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 3986362716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 3986362716}]}, 'public_addr': '172.21.15.154:6808/3986362716', 'cluster_addr': '172.21.15.154:6809/3986362716', 'heartbeat_back_addr': '172.21.15.154:6811/3986362716', 'heartbeat_front_addr': '172.21.15.154:6810/3986362716', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ccfee49f-f394-4de0-8e30-853f38e31232', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1978982202}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 1978982202}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 1978982202}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1978982202}]}, 'public_addr': '172.21.15.204:6808/1978982202', 'cluster_addr': '172.21.15.204:6809/1978982202', 'heartbeat_back_addr': '172.21.15.204:6811/1978982202', 'heartbeat_front_addr': '172.21.15.204:6810/1978982202', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '03b1c505-1b92-4daa-9da4-c254be2ba792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 3608171775}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 3608171775}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6816', 'nonce': 3608171775}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6815', 'nonce': 3608171775}]}, 'public_addr': '172.21.15.38:6813/3608171775', 'cluster_addr': '172.21.15.38:6814/3608171775', 'heartbeat_back_addr': '172.21.15.38:6816/3608171775', 'heartbeat_front_addr': '172.21.15.38:6815/3608171775', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '8fbee7a0-2ee6-4551-bf06-4e013e413fb4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 2580659844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6805', 'nonce': 2580659844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6807', 'nonce': 2580659844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 2580659844}]}, 'public_addr': '172.21.15.118:6804/2580659844', 'cluster_addr': '172.21.15.118:6805/2580659844', 'heartbeat_back_addr': '172.21.15.118:6807/2580659844', 'heartbeat_front_addr': '172.21.15.118:6806/2580659844', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '0808cb41-f62e-49b0-935d-616bea1a4db5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 2352891695}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 2352891695}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 2352891695}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 2352891695}]}, 'public_addr': '172.21.15.154:6812/2352891695', 'cluster_addr': '172.21.15.154:6813/2352891695', 'heartbeat_back_addr': '172.21.15.154:6815/2352891695', 'heartbeat_front_addr': '172.21.15.154:6814/2352891695', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '427dd574-f2a8-49e5-a32a-cfda023a514a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 2304836889}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 2304836889}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 2304836889}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 2304836889}]}, 'public_addr': '172.21.15.204:6800/2304836889', 'cluster_addr': '172.21.15.204:6801/2304836889', 'heartbeat_back_addr': '172.21.15.204:6803/2304836889', 'heartbeat_front_addr': '172.21.15.204:6802/2304836889', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '17942689-1130-4b6b-9f98-1605a03731b1', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 58, 'down_at': 59, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6809', 'nonce': 1688419254}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6810', 'nonce': 1688419254}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 1688419254}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6811', 'nonce': 1688419254}]}, 'public_addr': '172.21.15.38:6809/1688419254', 'cluster_addr': '172.21.15.38:6810/1688419254', 'heartbeat_back_addr': '172.21.15.38:6812/1688419254', 'heartbeat_front_addr': '172.21.15.38:6811/1688419254', 'state': ['exists']}, {'osd': 13, 'uuid': '62161cab-6d14-4571-971d-217601e38fbc', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 3727425991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6813', 'nonce': 3727425991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6815', 'nonce': 3727425991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 3727425991}]}, 'public_addr': '172.21.15.118:6812/3727425991', 'cluster_addr': '172.21.15.118:6813/3727425991', 'heartbeat_back_addr': '172.21.15.118:6815/3727425991', 'heartbeat_front_addr': '172.21.15.118:6814/3727425991', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '559ef141-27db-452d-9577-a6135421f64f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 538351706}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 538351706}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 538351706}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 538351706}]}, 'public_addr': '172.21.15.154:6804/538351706', 'cluster_addr': '172.21.15.154:6805/538351706', 'heartbeat_back_addr': '172.21.15.154:6807/538351706', 'heartbeat_front_addr': '172.21.15.154:6806/538351706', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '98f9ceba-79df-410a-9788-393057e79c30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 501019151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 501019151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 501019151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 501019151}]}, 'public_addr': '172.21.15.204:6812/501019151', 'cluster_addr': '172.21.15.204:6813/501019151', 'heartbeat_back_addr': '172.21.15.204:6815/501019151', 'heartbeat_front_addr': '172.21.15.204:6814/501019151', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:03.855267+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:32:00.915157+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:32:00.198442+0000', 'dead_epoch': 41}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:04.310901+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:03.659118+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:33:09.623455+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.808086+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.499908+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:07.828658+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:32:00.655711+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:32:01.092457+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:32:00.246352+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:32:01.027609+0000', 'dead_epoch': 59}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.963999+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.534738+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:02.472566+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.3', 'osds': [7, 15, 13]}, {'pgid': '4.2', 'osds': [3, 12, 9]}, {'pgid': '4.4', 'osds': [3, 8, 5]}, {'pgid': '4.11', 'osds': [7, 3, 15]}, {'pgid': '4.12', 'osds': [14, 8, 3]}, {'pgid': '4.13', 'osds': [7, 14, 6]}, {'pgid': '4.14', 'osds': [4, 7, 3]}, {'pgid': '4.16', 'osds': [10, 6, 12]}, {'pgid': '4.19', 'osds': [6, 0, 7]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:33:20.835 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-13T11:33:20.837 INFO:tasks.rados.rados.0.smithi204.stdout:1453: finishing write tid 3 to smithi20438004-12 2024-09-13T11:33:20.837 INFO:tasks.rados.rados.0.smithi204.stdout:1457: got (0) Success 2024-09-13T11:33:20.837 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 111 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-09-13T11:33:20.837 INFO:tasks.rados.rados.0.smithi204.stdout:1453: finishing write tid 4 to smithi20438004-12 2024-09-13T11:33:20.837 INFO:tasks.rados.rados.0.smithi204.stdout:1453: oid 12 updating version 0 to 82 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1453: oid 12 version 82 is already newer than 81 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1453: oid 12 updating version 82 to 83 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 83 (ObjNum 290 snap 0 seq_num 290) dirty exists 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1453: left oid 12 (ObjNum 290 snap 0 seq_num 290) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1451: done (7 left) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1452: done (6 left) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1453: done (5 left) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1454: done (4 left) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1455: done (3 left) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1456: done (2 left) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1457: done (1 left) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1459: read oid 35 snap -1 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1459: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:20.838 INFO:tasks.rados.rados.0.smithi204.stdout:1460: tier_evict oid 44 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1460: tier_evict oid 44 snap -1 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1458: got (22) Invalid argument 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1458: done (2 left) 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1461: delete oid 23 current snap is 0 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1460: got (22) Invalid argument 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1460: done (2 left) 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1461: done (1 left) 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1462: write oid 6 current snap is 0 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1462: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1462: writing smithi20438004-6 from 507644 to 1301766 tid 1 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1462: writing smithi20438004-6 from 1920742 to 2543827 tid 2 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1463: read oid 3 snap -1 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1463: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1464: delete oid 32 current snap is 0 2024-09-13T11:33:20.839 INFO:tasks.rados.rados.0.smithi204.stdout:1459: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-13T11:33:20.840 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:20.705+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:20.840 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:20.841 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:20.836+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:1462: finishing write tid 1 to smithi20438004-6 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:1462: finishing write tid 2 to smithi20438004-6 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:1462: finishing write tid 3 to smithi20438004-6 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:1462: finishing write tid 4 to smithi20438004-6 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:1462: oid 6 updating version 0 to 73 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:1462: oid 6 version 73 is already newer than 72 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:1462: oid 6 version 73 is already newer than 71 2024-09-13T11:33:20.848 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 73 (ObjNum 292 snap 0 seq_num 292) dirty exists 2024-09-13T11:33:20.849 INFO:tasks.rados.rados.0.smithi204.stdout:1462: left oid 6 (ObjNum 292 snap 0 seq_num 292) 2024-09-13T11:33:20.849 INFO:tasks.rados.rados.0.smithi204.stdout:1463: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-13T11:33:20.857 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2488707 2024-09-13T11:33:20.864 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:20.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:20.916 INFO:tasks.rados.rados.0.smithi204.stdout:1459: done (3 left) 2024-09-13T11:33:20.916 INFO:tasks.rados.rados.0.smithi204.stdout:1462: done (2 left) 2024-09-13T11:33:20.916 INFO:tasks.rados.rados.0.smithi204.stdout:1463: done (1 left) 2024-09-13T11:33:20.916 INFO:tasks.rados.rados.0.smithi204.stdout:1464: done (0 left) 2024-09-13T11:33:20.916 INFO:tasks.rados.rados.0.smithi204.stdout:1465: set_chunk oid 30 target oid 2024-09-13T11:33:20.923 INFO:tasks.rados.rados.0.smithi204.stdout:1465: set_chunk oid 30 offset: 167936 length: 742400 target oid c041fa01367e881f0f17f52f7dd517a2be28ca0c0ab7ad63f5884d7c73f01a0f offset: 0 snap -1 2024-09-13T11:33:20.923 INFO:tasks.rados.rados.0.smithi204.stdout:1466: write oid 32 current snap is 0 2024-09-13T11:33:20.923 INFO:tasks.rados.rados.0.smithi204.stdout:1466: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-13T11:33:20.929 INFO:tasks.rados.rados.0.smithi204.stdout:1466: writing smithi20438004-32 from 541258 to 1085878 tid 1 2024-09-13T11:33:20.936 INFO:tasks.rados.rados.0.smithi204.stdout:1466: writing smithi20438004-32 from 1637706 to 2264460 tid 2 2024-09-13T11:33:20.937 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:20.935+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:20.940 INFO:tasks.rados.rados.0.smithi204.stdout:1466: writing smithi20438004-32 from 2840948 to 2941258 tid 3 2024-09-13T11:33:20.940 INFO:tasks.rados.rados.0.smithi204.stdout:1467: write oid 44 current snap is 0 2024-09-13T11:33:20.940 INFO:tasks.rados.rados.0.smithi204.stdout:1465: finishing set_chunk to oid 30 2024-09-13T11:33:20.940 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 74 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-13T11:33:20.940 INFO:tasks.rados.rados.0.smithi204.stdout:1467: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-13T11:33:20.946 INFO:tasks.rados.rados.0.smithi204.stdout:1467: writing smithi20438004-44 from 558065 to 1177934 tid 1 2024-09-13T11:33:20.947 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:20.947 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:20.947 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:20.947 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:20.951 INFO:tasks.rados.rados.0.smithi204.stdout:1467: writing smithi20438004-44 from 1579835 to 2066600 tid 2 2024-09-13T11:33:20.956 INFO:tasks.rados.rados.0.smithi204.stdout:1467: writing smithi20438004-44 from 2468371 to 2958065 tid 3 2024-09-13T11:33:20.957 INFO:tasks.rados.rados.0.smithi204.stdout:1465: done (2 left) 2024-09-13T11:33:20.957 INFO:tasks.rados.rados.0.smithi204.stdout:1468: read oid 40 snap -1 2024-09-13T11:33:20.957 INFO:tasks.rados.rados.0.smithi204.stdout:1468: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:20.957 INFO:tasks.rados.rados.0.smithi204.stdout:1469: write oid 41 current snap is 0 2024-09-13T11:33:20.957 INFO:tasks.rados.rados.0.smithi204.stdout:1469: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-13T11:33:20.958 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:20.965 INFO:tasks.rados.rados.0.smithi204.stdout:1469: writing smithi20438004-41 from 574872 to 1269990 tid 1 2024-09-13T11:33:20.972 INFO:tasks.rados.rados.0.smithi204.stdout:1469: writing smithi20438004-41 from 2038317 to 2668740 tid 2 2024-09-13T11:33:20.973 INFO:tasks.rados.rados.0.smithi204.stdout:1470: read oid 33 snap -1 2024-09-13T11:33:20.973 INFO:tasks.rados.rados.0.smithi204.stdout:1470: expect deleted 2024-09-13T11:33:20.973 INFO:tasks.rados.rados.0.smithi204.stdout:1471: set_chunk oid 5 target oid 2024-09-13T11:33:20.979 INFO:tasks.rados.rados.0.smithi204.stdout:1471: set_chunk oid 5 offset: 800256 length: 679936 target oid 409636359aa79479967a2721ebc4c356ec379de69cc73ce84e1b3a5668df6ef7 offset: 0 snap -1 2024-09-13T11:33:20.979 INFO:tasks.rados.rados.0.smithi204.stdout:1472: read oid 3 snap -1 2024-09-13T11:33:20.979 INFO:tasks.rados.rados.0.smithi204.stdout:1472: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-13T11:33:20.979 INFO:tasks.rados.rados.0.smithi204.stdout:1473: tier_evict oid 43 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1473: tier_evict oid 43 snap -1 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: finishing write tid 1 to smithi20438004-32 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: finishing write tid 2 to smithi20438004-32 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: finishing write tid 3 to smithi20438004-32 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: finishing write tid 4 to smithi20438004-32 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: finishing write tid 5 to smithi20438004-32 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: oid 32 updating version 0 to 69 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: oid 32 version 69 is already newer than 68 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: oid 32 version 69 is already newer than 67 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: oid 32 version 69 is already newer than 66 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 69 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1466: left oid 32 (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:20.980 INFO:tasks.rados.rados.0.smithi204.stdout:1467: finishing write tid 1 to smithi20438004-44 2024-09-13T11:33:20.981 INFO:tasks.rados.rados.0.smithi204.stdout:1467: finishing write tid 2 to smithi20438004-44 2024-09-13T11:33:20.981 INFO:tasks.rados.rados.0.smithi204.stdout:1467: finishing write tid 3 to smithi20438004-44 2024-09-13T11:33:20.981 INFO:tasks.rados.rados.0.smithi204.stdout:1467: finishing write tid 4 to smithi20438004-44 2024-09-13T11:33:20.981 INFO:tasks.rados.rados.0.smithi204.stdout:1466: done (7 left) 2024-09-13T11:33:20.981 INFO:tasks.rados.rados.0.smithi204.stdout:1474: read oid 27 snap -1 2024-09-13T11:33:20.981 INFO:tasks.rados.rados.0.smithi204.stdout:1474: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-13T11:33:20.981 INFO:tasks.rados.rados.0.smithi204.stdout:1475: set_chunk oid 17 target oid 2024-09-13T11:33:21.001 INFO:tasks.rados.rados.0.smithi204.stdout:1475: set_chunk oid 17 offset: 132096 length: 2702336 target oid d9cc040a38c26b380f5dfe056e62ab5a8c8c018f04fa9adcb04562e2fd06b06f offset: 0 snap -1 2024-09-13T11:33:21.001 INFO:tasks.rados.rados.0.smithi204.stdout:1476: read oid 50 snap -1 2024-09-13T11:33:21.001 INFO:tasks.rados.rados.0.smithi204.stdout:1476: expect deleted 2024-09-13T11:33:21.001 INFO:tasks.rados.rados.0.smithi204.stdout:1477: set_chunk oid 39 target oid 2024-09-13T11:33:21.009 INFO:tasks.rados.rados.0.smithi204.stdout:1477: set_chunk oid 39 offset: 513024 length: 1068544 target oid 4ab4544b543eb3fe2e3aa33a38badcebc7bb38706766a1fec9ddd52a4f21cc7d offset: 0 snap -1 2024-09-13T11:33:21.009 INFO:tasks.rados.rados.0.smithi204.stdout:1478: read oid 29 snap -1 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1478: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1479: delete oid 25 current snap is 0 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1467: finishing write tid 5 to smithi20438004-44 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1467: oid 44 updating version 0 to 61 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1467: oid 44 version 61 is already newer than 60 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1467: oid 44 version 61 is already newer than 58 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1467: oid 44 version 61 is already newer than 59 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 61 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1467: left oid 44 (ObjNum 295 snap 0 seq_num 295) 2024-09-13T11:33:21.010 INFO:tasks.rados.rados.0.smithi204.stdout:1468: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:21.038 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:21.035+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:21.041 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:21.041 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:21.058 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:21.059 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:21.059 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:21.077 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:21.077 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:21.081 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:21.081 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:21.081 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:21.081 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:21.081 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:21.090 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:21.093 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:21.094 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:21.111 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:21.111 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:21.129 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:21.129 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:21.139 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:21.137+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:21.145 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:21.146 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:21.161 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:21.162 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: finishing write tid 1 to smithi20438004-41 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: finishing write tid 2 to smithi20438004-41 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: finishing write tid 3 to smithi20438004-41 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: finishing write tid 4 to smithi20438004-41 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: oid 41 updating version 0 to 119 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: oid 41 version 119 is already newer than 117 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: oid 41 version 119 is already newer than 118 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 119 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1469: left oid 41 (ObjNum 296 snap 0 seq_num 296) 2024-09-13T11:33:21.174 INFO:tasks.rados.rados.0.smithi204.stdout:1467: done (12 left) 2024-09-13T11:33:21.175 INFO:tasks.rados.rados.0.smithi204.stdout:1468: done (11 left) 2024-09-13T11:33:21.175 INFO:tasks.rados.rados.0.smithi204.stdout:1469: done (10 left) 2024-09-13T11:33:21.175 INFO:tasks.rados.rados.0.smithi204.stdout:1470: done (9 left) 2024-09-13T11:33:21.175 INFO:tasks.rados.rados.0.smithi204.stdout:1479: done (8 left) 2024-09-13T11:33:21.175 INFO:tasks.rados.rados.0.smithi204.stdout:1480: read oid 12 snap -1 2024-09-13T11:33:21.175 INFO:tasks.rados.rados.0.smithi204.stdout:1480: expect (ObjNum 290 snap 0 seq_num 290) 2024-09-13T11:33:21.175 INFO:tasks.rados.rados.0.smithi204.stdout:1481: set_chunk oid 1 target oid 2024-09-13T11:33:21.178 INFO:tasks.rados.rados.0.smithi204.stdout:1481: set_chunk oid 1 offset: 531968 length: 432128 target oid 6d1da7865efd14016755e89d35e3d3483169c35708edb0110137feb1e461f19c offset: 0 snap -1 2024-09-13T11:33:21.178 INFO:tasks.rados.rados.0.smithi204.stdout:1482: write (excl) oid 38 current snap is 0 2024-09-13T11:33:21.178 INFO:tasks.rados.rados.0.smithi204.stdout:1473: got (22) Invalid argument 2024-09-13T11:33:21.179 INFO:tasks.rados.rados.0.smithi204.stdout:1474: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-13T11:33:21.179 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:21.180 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:21.180 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:21.197 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:21.197 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:21.202 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:21.214 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:21.214 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:21.214 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:21.232 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:21.232 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:21.235 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1877532 2024-09-13T11:33:21.237 INFO:tasks.rados.rados.0.smithi204.stdout:1471: finishing set_chunk to oid 5 2024-09-13T11:33:21.238 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 74 (ObjNum 286 snap 0 seq_num 286) dirty exists 2024-09-13T11:33:21.238 INFO:tasks.rados.rados.0.smithi204.stdout:1472: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-13T11:33:21.240 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:21.240 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.145362317999997, 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.241 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.242 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.242 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:21.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.135087751, 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.246 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:33:21.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.12241985, 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.254 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:21.255 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.115810003, 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:33:21.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:21.260 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.107432973000002, 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:21.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:33:21.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.088170925999997, 2024-09-13T11:33:21.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010735344000000001, 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148570+0000", 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148571+0000", 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154390+0000", 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058191809999999997 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154404+0000", 2024-09-13T11:33:21.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4584e-05 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154415+0000", 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0661e-05 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154478+0000", 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2528999999999997e-05 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.268 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154502+0000", 2024-09-13T11:33:21.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4460999999999998e-05 2024-09-13T11:33:21.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159243+0000", 2024-09-13T11:33:21.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047404259999999998 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.159305+0000", 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2582999999999994e-05 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.082301794000003, 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:21.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.279 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.073400516, 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.280 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.281 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:33:21.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.284 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.065802793000003, 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.288 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.060398864, 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:33:21.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:33:21.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:21.293 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.047775174000002, 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:33:21.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.033595724000001, 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:33:21.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:33:21.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 54.025377556000002, 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.302 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:21.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:33:21.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:33:21.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:21.308 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.994554153000003, 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:21.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.314 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.847839366000002, 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.318 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 45.142667314000001, 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:21.319 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:33:21.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:21.323 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.394714944, 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013151014000000001, 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:21.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842045+0000", 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865e-05 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848182+0000", 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061372370000000002 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848205+0000", 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.298e-05 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848218+0000", 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.348e-05 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848300+0000", 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1766000000000003e-05 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848327+0000", 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6446999999999999e-05 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855119+0000", 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067926490000000004 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855177+0000", 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7590000000000003e-05 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.327 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1458 4.e 4:7984cf84:::smithi20438004-46:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 44.209102891999997, 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012793834, 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1458 2024-09-13T11:33:21.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027640+0000", 2024-09-13T11:33:21.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.046e-06 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027651+0000", 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.102e-05 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027669+0000", 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027681+0000", 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2177999999999999e-05 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027762+0000", 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0331e-05 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027868+0000", 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106443 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.040432+0000", 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012563632999999999 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1787 4.e 4:7022fc62:::smithi20438004-6:head [read 188928~1265152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.963010201000003, 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013752110999999999, 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.332 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1787 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273731+0000", 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273734+0000", 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3510000000000002e-06 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273741+0000", 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273787+0000", 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6198000000000001e-05 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274432+0000", 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064478300000000001 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274461+0000", 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9051999999999999e-05 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281751+0000", 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007289963 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281984+0000", 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023257099999999999 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.287483+0000", 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054989829999999998 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2762 4.e 4:7984cf84:::smithi20438004-46:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e66)", 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.340552783, 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022335361000000001, 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:21.337 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2762 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896189+0000", 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3519999999999999e-06 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896196+0000", 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:21.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896210+0000", 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3439e-05 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896217+0000", 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5889999999999997e-06 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896278+0000", 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1072000000000001e-05 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896338+0000", 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9998999999999999e-05 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.918523+0000", 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022185099999999999 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:21.341 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:21.342 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:21.343 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:21.239+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:21.343 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:21.344 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:21.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:1475: finishing set_chunk to oid 17 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 80 (ObjNum 289 snap 0 seq_num 289) dirty exists 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:1477: finishing set_chunk to oid 39 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 513024 length: 1068544 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 0 (ObjNum 247 snap 0 seq_num 247) dirty exists 2024-09-13T11:33:21.345 INFO:tasks.rados.rados.0.smithi204.stdout:1478: expect (ObjNum 218 snap 0 seq_num 218) 2024-09-13T11:33:21.346 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:21.340+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:21.346 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:21.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:21.368 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:21.432 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 32972 2024-09-13T11:33:21.444 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:21.443+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:21.493 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:21.494 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:21.497 INFO:tasks.rados.rados.0.smithi204.stdout:1480: expect (ObjNum 290 snap 0 seq_num 290) 2024-09-13T11:33:21.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:21.545 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:21.544+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:21.571 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1771839 2024-09-13T11:33:21.614 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2055918 2024-09-13T11:33:21.629 INFO:tasks.rados.rados.0.smithi204.stdout:1481: finishing set_chunk to oid 1 2024-09-13T11:33:21.630 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 66 (ObjNum 241 snap 0 seq_num 241) dirty exists 2024-09-13T11:33:21.630 INFO:tasks.rados.rados.0.smithi204.stdout:1482: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-13T11:33:21.635 INFO:tasks.rados.rados.0.smithi204.stdout:1482: writing smithi20438004-38 from 608486 to 1170455 tid 1 2024-09-13T11:33:21.643 INFO:tasks.rados.rados.0.smithi204.stdout:1482: writing smithi20438004-38 from 1871634 to 2505726 tid 2 2024-09-13T11:33:21.645 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:21.643+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:21.646 INFO:tasks.rados.rados.0.smithi204.stdout:1482: writing smithi20438004-38 from 2950640 to 3008486 tid 3 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1471: done (10 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1472: done (9 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1473: done (8 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1474: done (7 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1475: done (6 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1476: done (5 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1477: done (4 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1478: done (3 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1480: done (2 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1481: done (1 left) 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1483: write (excl) oid 36 current snap is 0 2024-09-13T11:33:21.647 INFO:tasks.rados.rados.0.smithi204.stdout:1483: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-13T11:33:21.649 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:21.649 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:21.653 INFO:tasks.rados.rados.0.smithi204.stdout:1483: writing smithi20438004-36 from 625293 to 1262511 tid 1 2024-09-13T11:33:21.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:21.659 INFO:tasks.rados.rados.0.smithi204.stdout:1483: writing smithi20438004-36 from 1813763 to 2307866 tid 2 2024-09-13T11:33:21.664 INFO:tasks.rados.rados.0.smithi204.stdout:1483: writing smithi20438004-36 from 2978063 to 3025293 tid 3 2024-09-13T11:33:21.664 INFO:tasks.rados.rados.0.smithi204.stdout:1484: tier_promote oid 10 2024-09-13T11:33:21.664 INFO:tasks.rados.rados.0.smithi204.stdout:1482: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:21.664 INFO:tasks.rados.rados.0.smithi204.stdout:1485: read oid 22 snap -1 2024-09-13T11:33:21.664 INFO:tasks.rados.rados.0.smithi204.stdout:1485: expect (ObjNum 285 snap 0 seq_num 285) 2024-09-13T11:33:21.664 INFO:tasks.rados.rados.0.smithi204.stdout:1486: read oid 1 snap -1 2024-09-13T11:33:21.664 INFO:tasks.rados.rados.0.smithi204.stdout:1486: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1487: read oid 30 snap -1 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1487: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1488: write oid 18 current snap is 0 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: oid 38 updating version 0 to 121 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: oid 38 updating version 121 to 122 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: oid 38 updating version 122 to 124 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: oid 38 version 124 is already newer than 123 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 124 (ObjNum 298 snap 0 seq_num 298) dirty exists 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1482: left oid 38 (ObjNum 298 snap 0 seq_num 298) 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1483: finishing write tid 1 to smithi20438004-36 2024-09-13T11:33:21.665 INFO:tasks.rados.rados.0.smithi204.stdout:1488: seq_num 300 ranges {642100=712467,1872245=637761} 2024-09-13T11:33:21.672 INFO:tasks.rados.rados.0.smithi204.stdout:1488: writing smithi20438004-18 from 642100 to 1354567 tid 1 2024-09-13T11:33:21.681 INFO:tasks.rados.rados.0.smithi204.stdout:1488: writing smithi20438004-18 from 1872245 to 2510006 tid 2 2024-09-13T11:33:21.681 INFO:tasks.rados.rados.0.smithi204.stdout:1482: done (6 left) 2024-09-13T11:33:21.681 INFO:tasks.rados.rados.0.smithi204.stdout:1489: set_chunk oid 38 target oid 2024-09-13T11:33:21.684 INFO:tasks.rados.rados.0.smithi204.stdout:1489: set_chunk oid 38 offset: 2347008 length: 160256 target oid 163c89c573e50611389b7c37f4ecb317540e27c5ae791b62761cb29dbaa5441a offset: 0 snap -1 2024-09-13T11:33:21.684 INFO:tasks.rados.rados.0.smithi204.stdout:1490: read oid 24 snap -1 2024-09-13T11:33:21.685 INFO:tasks.rados.rados.0.smithi204.stdout:1490: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-13T11:33:21.685 INFO:tasks.rados.rados.0.smithi204.stdout:1491: read oid 34 snap -1 2024-09-13T11:33:21.685 INFO:tasks.rados.rados.0.smithi204.stdout:1491: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:21.685 INFO:tasks.rados.rados.0.smithi204.stdout:1492: write (excl) oid 15 current snap is 0 2024-09-13T11:33:21.685 INFO:tasks.rados.rados.0.smithi204.stdout:1492: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-13T11:33:21.693 INFO:tasks.rados.rados.0.smithi204.stdout:1492: writing smithi20438004-15 from 658907 to 1446623 tid 1 2024-09-13T11:33:21.698 INFO:tasks.rados.rados.0.smithi204.stdout:1492: writing smithi20438004-15 from 1930727 to 2428499 tid 2 2024-09-13T11:33:21.702 INFO:tasks.rados.rados.0.smithi204.stdout:1492: writing smithi20438004-15 from 3032909 to 3058907 tid 3 2024-09-13T11:33:21.702 INFO:tasks.rados.rados.0.smithi204.stdout:1493: set_chunk oid 8 target oid 2024-09-13T11:33:21.707 INFO:tasks.rados.rados.0.smithi204.stdout:1493: set_chunk oid 8 offset: 1006080 length: 316928 target oid 97a24a79b391f3572206423a136b369c782d829b84d69e030c70268b8dea62b1 offset: 0 snap -1 2024-09-13T11:33:21.707 INFO:tasks.rados.rados.0.smithi204.stdout:1494: set_chunk oid 5 target oid 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1494: set_chunk oid 5 offset: 1085952 length: 433152 target oid acdf8837f6999872e6975a28a8ecc988a4a4a4697db98effd93625ebdd1584eb offset: 0 snap -1 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1495: read oid 2 snap -1 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1495: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1496: read oid 19 snap -1 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1496: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1497: read oid 41 snap -1 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1497: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1498: read oid 4 snap -1 2024-09-13T11:33:21.714 INFO:tasks.rados.rados.0.smithi204.stdout:1498: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-13T11:33:21.715 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:21.715 INFO:tasks.rados.rados.0.smithi204.stdout:1486: expect (ObjNum 241 snap 0 seq_num 241) 2024-09-13T11:33:21.719 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 626041 2024-09-13T11:33:21.746 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:21.745+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:21.767 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:21.767 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:21.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:21.805 INFO:tasks.rados.rados.0.smithi204.stdout:1483: finishing write tid 2 to smithi20438004-36 2024-09-13T11:33:21.805 INFO:tasks.rados.rados.0.smithi204.stdout:1483: finishing write tid 3 to smithi20438004-36 2024-09-13T11:33:21.805 INFO:tasks.rados.rados.0.smithi204.stdout:1487: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-13T11:33:21.847 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:21.847+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:21.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1536516 2024-09-13T11:33:21.880 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 74408 2024-09-13T11:33:21.886 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:21.886 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:21.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:21.924 INFO:tasks.rados.rados.0.smithi204.stdout:1483: finishing write tid 4 to smithi20438004-36 2024-09-13T11:33:21.924 INFO:tasks.rados.rados.0.smithi204.stdout:1483: finishing write tid 5 to smithi20438004-36 2024-09-13T11:33:21.924 INFO:tasks.rados.rados.0.smithi204.stdout:1483: oid 36 updating version 0 to 62 2024-09-13T11:33:21.925 INFO:tasks.rados.rados.0.smithi204.stdout:1483: oid 36 updating version 62 to 63 2024-09-13T11:33:21.925 INFO:tasks.rados.rados.0.smithi204.stdout:1483: oid 36 updating version 63 to 64 2024-09-13T11:33:21.925 INFO:tasks.rados.rados.0.smithi204.stdout:1483: oid 36 version 64 is already newer than 61 2024-09-13T11:33:21.925 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 64 (ObjNum 299 snap 0 seq_num 299) dirty exists 2024-09-13T11:33:21.925 INFO:tasks.rados.rados.0.smithi204.stdout:1483: left oid 36 (ObjNum 299 snap 0 seq_num 299) 2024-09-13T11:33:21.925 INFO:tasks.rados.rados.0.smithi204.stdout:1485: expect (ObjNum 285 snap 0 seq_num 285) 2024-09-13T11:33:21.925 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:21.948 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:21.947+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:22.043 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.044 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.049 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:22.048+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:22.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:22.066 INFO:tasks.rados.rados.0.smithi204.stdout:1484: got (0) Success 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 114 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1488: finishing write tid 1 to smithi20438004-18 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1483: done (15 left) 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1484: done (14 left) 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1485: done (13 left) 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1486: done (12 left) 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1487: done (11 left) 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1499: read oid 40 snap -1 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1499: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1500: tier_evict oid 22 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1500: tier_evict oid 22 snap -1 2024-09-13T11:33:22.067 INFO:tasks.rados.rados.0.smithi204.stdout:1489: finishing set_chunk to oid 38 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 124 (ObjNum 298 snap 0 seq_num 298) dirty exists 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1488: finishing write tid 2 to smithi20438004-18 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1488: finishing write tid 3 to smithi20438004-18 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1489: done (12 left) 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1501: write oid 27 current snap is 0 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1488: finishing write tid 4 to smithi20438004-18 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1488: oid 18 updating version 0 to 122 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1488: oid 18 version 122 is already newer than 120 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1488: oid 18 version 122 is already newer than 121 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 122 (ObjNum 300 snap 0 seq_num 300) dirty exists 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1488: left oid 18 (ObjNum 300 snap 0 seq_num 300) 2024-09-13T11:33:22.068 INFO:tasks.rados.rados.0.smithi204.stdout:1490: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-13T11:33:22.150 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:22.148+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:22.176 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:22.201 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.201 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:22.231 INFO:tasks.rados.rados.0.smithi204.stdout:1491: expect (ObjNum 213 snap 0 seq_num 213) 2024-09-13T11:33:22.251 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:22.249+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:22.309 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:22.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:33:22.319 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.319 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:22.353 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:22.351+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:22.384 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:22.454 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:22.451+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:22.460 INFO:tasks.rados.rados.0.smithi204.stdout:1492: finishing write tid 1 to smithi20438004-15 2024-09-13T11:33:22.460 INFO:tasks.rados.rados.0.smithi204.stdout:1492: finishing write tid 2 to smithi20438004-15 2024-09-13T11:33:22.460 INFO:tasks.rados.rados.0.smithi204.stdout:1492: finishing write tid 3 to smithi20438004-15 2024-09-13T11:33:22.460 INFO:tasks.rados.rados.0.smithi204.stdout:1492: finishing write tid 4 to smithi20438004-15 2024-09-13T11:33:22.460 INFO:tasks.rados.rados.0.smithi204.stdout:1492: finishing write tid 5 to smithi20438004-15 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1492: oid 15 updating version 0 to 87 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1492: oid 15 version 87 is already newer than 86 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1492: oid 15 updating version 87 to 88 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1492: oid 15 updating version 88 to 89 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 89 (ObjNum 301 snap 0 seq_num 301) dirty exists 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1492: left oid 15 (ObjNum 301 snap 0 seq_num 301) 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1493: finishing set_chunk to oid 8 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 35 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1494: finishing set_chunk to oid 5 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 1085952 length: 433152 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 286 snap 0 seq_num 286) dirty exists 2024-09-13T11:33:22.461 INFO:tasks.rados.rados.0.smithi204.stdout:1501: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-13T11:33:22.466 INFO:tasks.rados.rados.0.smithi204.stdout:1501: writing smithi20438004-27 from 675714 to 1138679 tid 1 2024-09-13T11:33:22.471 INFO:tasks.rados.rados.0.smithi204.stdout:1501: writing smithi20438004-27 from 1589209 to 2230639 tid 2 2024-09-13T11:33:22.476 INFO:tasks.rados.rados.0.smithi204.stdout:1501: writing smithi20438004-27 from 2943979 to 3075714 tid 3 2024-09-13T11:33:22.477 INFO:tasks.rados.rados.0.smithi204.stdout:1488: done (12 left) 2024-09-13T11:33:22.477 INFO:tasks.rados.rados.0.smithi204.stdout:1490: done (11 left) 2024-09-13T11:33:22.477 INFO:tasks.rados.rados.0.smithi204.stdout:1491: done (10 left) 2024-09-13T11:33:22.477 INFO:tasks.rados.rados.0.smithi204.stdout:1492: done (9 left) 2024-09-13T11:33:22.477 INFO:tasks.rados.rados.0.smithi204.stdout:1493: done (8 left) 2024-09-13T11:33:22.477 INFO:tasks.rados.rados.0.smithi204.stdout:1494: done (7 left) 2024-09-13T11:33:22.477 INFO:tasks.rados.rados.0.smithi204.stdout:1502: set_chunk oid 15 target oid 2024-09-13T11:33:22.493 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.493 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.495 INFO:tasks.rados.rados.0.smithi204.stdout:1502: set_chunk oid 15 offset: 1072128 length: 1526784 target oid 394a8b4ef5ada98d36bb5703fc5f69d8725d319ce958685894d78867a5059f0a offset: 0 snap -1 2024-09-13T11:33:22.495 INFO:tasks.rados.rados.0.smithi204.stdout:1503: write (excl) oid 12 current snap is 0 2024-09-13T11:33:22.495 INFO:tasks.rados.rados.0.smithi204.stdout:1503: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-13T11:33:22.501 INFO:tasks.rados.rados.0.smithi204.stdout:1503: writing smithi20438004-12 from 692521 to 1230735 tid 1 2024-09-13T11:33:22.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:22.510 INFO:tasks.rados.rados.0.smithi204.stdout:1503: writing smithi20438004-12 from 1931338 to 2716426 tid 2 2024-09-13T11:33:22.510 INFO:tasks.rados.rados.0.smithi204.stdout:1504: read oid 31 snap -1 2024-09-13T11:33:22.510 INFO:tasks.rados.rados.0.smithi204.stdout:1504: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:33:22.510 INFO:tasks.rados.rados.0.smithi204.stdout:1505: write oid 16 current snap is 0 2024-09-13T11:33:22.510 INFO:tasks.rados.rados.0.smithi204.stdout:1497: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-13T11:33:22.555 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:22.553+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:22.608 INFO:tasks.rados.rados.0.smithi204.stdout:1495: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-13T11:33:22.613 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.613 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:22.641 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2128179 2024-09-13T11:33:22.655 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:22.653+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:22.738 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.738 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.747 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:22.756 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:22.755+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:22.789 INFO:tasks.rados.rados.0.smithi204.stdout:1496: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:22.789 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:22.831 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 11966 2024-09-13T11:33:22.851 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.851 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.857 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:22.855+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:22.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:22.908 INFO:tasks.rados.rados.0.smithi204.stdout:1498: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-13T11:33:22.958 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:22.957+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:22.962 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:22.962 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:22.971 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:23.059 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:23.058+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:23.063 INFO:tasks.rados.rados.0.smithi204.stdout:1499: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:23.074 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:23.074 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:23.083 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:23.084 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1906640 2024-09-13T11:33:23.160 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:23.159+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:23.186 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:23.186 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:23.195 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:23.200 INFO:tasks.rados.rados.0.smithi204.stdout:1500: got (22) Invalid argument 2024-09-13T11:33:23.200 INFO:tasks.rados.rados.0.smithi204.stdout:1501: finishing write tid 1 to smithi20438004-27 2024-09-13T11:33:23.201 INFO:tasks.rados.rados.0.smithi204.stdout:1501: finishing write tid 2 to smithi20438004-27 2024-09-13T11:33:23.201 INFO:tasks.rados.rados.0.smithi204.stdout:1505: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-13T11:33:23.207 INFO:tasks.rados.rados.0.smithi204.stdout:1505: writing smithi20438004-16 from 709328 to 1322791 tid 1 2024-09-13T11:33:23.215 INFO:tasks.rados.rados.0.smithi204.stdout:1505: writing smithi20438004-16 from 1989820 to 2634919 tid 2 2024-09-13T11:33:23.215 INFO:tasks.rados.rados.0.smithi204.stdout:1495: done (10 left) 2024-09-13T11:33:23.215 INFO:tasks.rados.rados.0.smithi204.stdout:1496: done (9 left) 2024-09-13T11:33:23.215 INFO:tasks.rados.rados.0.smithi204.stdout:1497: done (8 left) 2024-09-13T11:33:23.215 INFO:tasks.rados.rados.0.smithi204.stdout:1498: done (7 left) 2024-09-13T11:33:23.215 INFO:tasks.rados.rados.0.smithi204.stdout:1499: done (6 left) 2024-09-13T11:33:23.216 INFO:tasks.rados.rados.0.smithi204.stdout:1500: done (5 left) 2024-09-13T11:33:23.216 INFO:tasks.rados.rados.0.smithi204.stdout:1506: set_chunk oid 3 target oid 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1506: set_chunk oid 3 offset: 1853440 length: 238592 target oid b29c93df0cffefba8dbd271a2b2dbf96f6bf1b7565ebbc57bf205af49fe80615 offset: 0 snap -1 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1507: read oid 25 snap -1 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1507: expect deleted 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1508: tier_promote oid 35 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1501: finishing write tid 3 to smithi20438004-27 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1509: tier_evict oid 20 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1509: tier_evict oid 20 snap -1 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1501: finishing write tid 4 to smithi20438004-27 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1501: finishing write tid 5 to smithi20438004-27 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1501: oid 27 updating version 0 to 102 2024-09-13T11:33:23.220 INFO:tasks.rados.rados.0.smithi204.stdout:1501: oid 27 version 102 is already newer than 100 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1501: oid 27 version 102 is already newer than 101 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1501: oid 27 version 102 is already newer than 99 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 102 (ObjNum 302 snap 0 seq_num 302) dirty exists 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1501: left oid 27 (ObjNum 302 snap 0 seq_num 302) 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1502: finishing set_chunk to oid 15 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 89 (ObjNum 301 snap 0 seq_num 301) dirty exists 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1503: finishing write tid 1 to smithi20438004-12 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1503: finishing write tid 2 to smithi20438004-12 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1503: finishing write tid 3 to smithi20438004-12 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1503: finishing write tid 4 to smithi20438004-12 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1503: oid 12 updating version 0 to 85 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1503: oid 12 updating version 85 to 86 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:1503: oid 12 version 86 is already newer than 84 2024-09-13T11:33:23.221 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 86 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-09-13T11:33:23.222 INFO:tasks.rados.rados.0.smithi204.stdout:1503: left oid 12 (ObjNum 303 snap 0 seq_num 303) 2024-09-13T11:33:23.222 INFO:tasks.rados.rados.0.smithi204.stdout:1504: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-13T11:33:23.262 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:23.260+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:23.311 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:33:23.339 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:23.340 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:23.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:23.349 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:23.363 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:23.362+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: finishing write tid 1 to smithi20438004-16 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: finishing write tid 2 to smithi20438004-16 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: finishing write tid 3 to smithi20438004-16 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: finishing write tid 4 to smithi20438004-16 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: oid 16 updating version 0 to 62 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: oid 16 updating version 62 to 64 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: oid 16 version 64 is already newer than 63 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 64 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1505: left oid 16 (ObjNum 304 snap 0 seq_num 304) 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1509: got (22) Invalid argument 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1501: done (8 left) 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1502: done (7 left) 2024-09-13T11:33:23.369 INFO:tasks.rados.rados.0.smithi204.stdout:1503: done (6 left) 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1504: done (5 left) 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1505: done (4 left) 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1507: done (3 left) 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1509: done (2 left) 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1510: read oid 40 snap -1 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1510: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1511: set_chunk oid 21 target oid 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1511: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1512: read oid 17 snap -1 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1512: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1513: set_chunk oid 50 target oid 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1513: set_chunk oid 50 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1514: write oid 22 current snap is 0 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:1506: finishing set_chunk to oid 3 2024-09-13T11:33:23.370 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 38 (ObjNum 268 snap 0 seq_num 268) dirty exists 2024-09-13T11:33:23.371 INFO:tasks.rados.rados.0.smithi204.stdout:1508: got (0) Success 2024-09-13T11:33:23.371 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 68 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-09-13T11:33:23.371 INFO:tasks.rados.rados.0.smithi204.stdout:1514: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-13T11:33:23.374 INFO:tasks.rados.rados.0.smithi204.stdout:1514: writing smithi20438004-22 from 726135 to 1131200 tid 1 2024-09-13T11:33:23.382 INFO:tasks.rados.rados.0.smithi204.stdout:1514: writing smithi20438004-22 from 1764655 to 2553412 tid 2 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1514: writing smithi20438004-22 from 3026248 to 3126135 tid 3 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1506: done (6 left) 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1508: done (5 left) 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1515: delete oid 18 current snap is 0 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1511: finishing set_chunk to oid 21 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1511: got expected ENOENT (src dne) 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1513: finishing set_chunk to oid 50 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1513: got expected ENOENT (src dne) 2024-09-13T11:33:23.387 INFO:tasks.rados.rados.0.smithi204.stdout:1512: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:23.452 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:23.452 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:23.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:23.464 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:23.463+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:23.506 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:23.506 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:23.525 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:23.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:23.525 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:23.541 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:23.557 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:23.557 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:23.563 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:23.563 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:23.565 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:23.563+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:23.565 INFO:tasks.rados.rados.0.smithi204.stdout:1510: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:23.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:23.573 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:23.573 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:23.580 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1271713 2024-09-13T11:33:23.591 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:23.591 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:23.607 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:23.607 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:23.623 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:23.623 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:23.624 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1279052 2024-09-13T11:33:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:23.640 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:23.652 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:23.657 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:23.657 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:23.665 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:23.664+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:23.674 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:24.318 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:24.318 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:24.319 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:24.319 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:24.319 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:24.319 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:24.321 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:24.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:206 3.1 e38/30 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.582417810999999, 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012415890000000001, 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092192+0000", 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.63e-07 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100358+0000", 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081664059999999993 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100384+0000", 2024-09-13T11:33:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5287e-05 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100401+0000", 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7744000000000002e-05 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100478+0000", 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6769999999999999e-05 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100508+0000", 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0134e-05 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104514+0000", 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040060590000000002 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104607+0000", 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3027000000000002e-05 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:201 3.1 e38/30 3:b265d015:::smithi20438004-2 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.574172472000001, 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012387119, 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100438+0000", 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1000000000000001e-06 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108964+0000", 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085255959999999999 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108984+0000", 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0466e-05 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108996+0000", 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2149e-05 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109097+0000", 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010042799999999999 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109121+0000", 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.455e-05 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112773+0000", 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036519780000000002 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112824+0000", 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0852e-05 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.565551001000003, 2024-09-13T11:33:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096300529999999995, 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109060+0000", 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116086+0000", 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070260239999999996 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116100+0000", 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4209999999999999e-05 2024-09-13T11:33:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116109+0000", 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6569999999999992e-06 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116160+0000", 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0470000000000003e-05 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116191+0000", 2024-09-13T11:33:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1494e-05 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118639+0000", 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024481030000000001 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118688+0000", 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8866999999999998e-05 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.558479206000001, 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010922463, 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116131+0000", 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6899999999999998e-07 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124407+0000", 2024-09-13T11:33:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082764919999999999 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124422+0000", 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5404999999999999e-05 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124432+0000", 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8910000000000007e-06 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124485+0000", 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2750999999999997e-05 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124508+0000", 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3076999999999999e-05 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127004+0000", 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024962930000000001 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127053+0000", 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8084999999999998e-05 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.544815108000002, 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091903760000000001, 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129795+0000", 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8700000000000001e-07 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135325+0000", 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055302190000000003 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135340+0000", 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135351+0000", 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1358999999999999e-05 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135418+0000", 2024-09-13T11:33:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000002e-05 2024-09-13T11:33:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135447+0000", 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8415000000000002e-05 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138936+0000", 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034896800000000002 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138985+0000", 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8338999999999999e-05 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.534620036, 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010981263, 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139990+0000", 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2e-07 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147357+0000", 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073669950000000003 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147376+0000", 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9453000000000002e-05 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147388+0000", 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1527e-05 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147472+0000", 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4630999999999997e-05 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147493+0000", 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0687e-05 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150899+0000", 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034054739999999999 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150971+0000", 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1976000000000003e-05 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.527169028000003, 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015742025999999999, 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147441+0000", 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7700000000000004e-07 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157019+0000", 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095782090000000007 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157057+0000", 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157072+0000", 2024-09-13T11:33:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5082999999999999e-05 2024-09-13T11:33:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157157+0000", 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5065e-05 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157188+0000", 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0920000000000002e-05 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163133+0000", 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059446890000000004 2024-09-13T11:33:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163182+0000", 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9564999999999997e-05 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.517096940000002, 2024-09-13T11:33:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015556076, 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157514+0000", 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167722+0000", 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010208016 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167740+0000", 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7668000000000001e-05 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167754+0000", 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4647999999999999e-05 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167831+0000", 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7229000000000002e-05 2024-09-13T11:33:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167855+0000", 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3467000000000001e-05 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173001+0000", 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051458160000000001 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173068+0000", 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7720999999999998e-05 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.506828448, 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011370243, 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3499999999999996e-07 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175562+0000", 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077811060000000003 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175578+0000", 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175587+0000", 2024-09-13T11:33:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:33:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175668+0000", 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1224999999999996e-05 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175695+0000", 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6505000000000001e-05 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179075+0000", 2024-09-13T11:33:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003380508 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179151+0000", 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5927000000000007e-05 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.499001786999997, 2024-09-13T11:33:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017514140000000001, 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188284+0000", 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012675702 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188299+0000", 2024-09-13T11:33:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5122e-05 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188308+0000", 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9819999999999997e-06 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188367+0000", 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8774999999999999e-05 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188387+0000", 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0649000000000002e-05 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193056+0000", 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046684830000000002 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193122+0000", 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5790999999999994e-05 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.486276809000003, 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015855429000000001, 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188333+0000", 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-07 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199600+0000", 2024-09-13T11:33:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011266717000000001 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199613+0000", 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3682e-05 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199620+0000", 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1339999999999998e-06 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199676+0000", 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5689000000000002e-05 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199708+0000", 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2082999999999998e-05 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204123+0000", 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004414734 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204188+0000", 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.474968271999998, 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092010560000000009, 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199642+0000", 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9599999999999999e-07 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205539+0000", 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058972290000000004 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205553+0000", 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4577e-05 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205564+0000", 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0254e-05 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205630+0000", 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6057999999999997e-05 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205658+0000", 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7979000000000001e-05 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208792+0000", 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031342499999999999 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208842+0000", 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0213000000000002e-05 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:245 3.1 e38/30 3:adf25eb4:::smithi20438004-46 low_tier:head v 38'15, mlcod=38'11)", 2024-09-13T11:33:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.469021943000001, 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011146911000000001, 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205588+0000", 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213344+0000", 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077554440000000002 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213359+0000", 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5455999999999999e-05 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213369+0000", 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9890000000000004e-06 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213453+0000", 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4153999999999994e-05 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213479+0000", 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5831999999999999e-05 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216694+0000", 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032149990000000001 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216734+0000", 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0102999999999999e-05 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:260 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.455250921999998, 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011239344, 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 260 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219360+0000", 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219366+0000", 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7179999999999999e-06 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219376+0000", 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9990000000000003e-06 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219382+0000", 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8150000000000002e-06 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219428+0000", 2024-09-13T11:33:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5751999999999998e-05 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4119000000000001e-05 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227887+0000", 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083953989999999996 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228998+0000", 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001110764 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229290+0000", 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029145400000000002 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230489+0000", 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001199786 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230548+0000", 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8900999999999997e-05 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230598+0000", 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9391999999999997e-05 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:249 3.1 e38/30 3:81faf980:::smithi20438004-50 low_tier:head v 38'21, mlcod=38'18)", 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.451974786000001, 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091134460000000007, 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.68e-07 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227843+0000", 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005207494 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227853+0000", 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117e-05 2024-09-13T11:33:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227860+0000", 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4880000000000001e-06 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227912+0000", 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1900000000000001e-05 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227933+0000", 2024-09-13T11:33:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0474000000000001e-05 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231709+0000", 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037762759999999999 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231748+0000", 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9029000000000002e-05 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:278 4.2 4:4d4c8512:::smithi20438004-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.443615186000002, 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091148359999999994, 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230995+0000", 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4200000000000004e-07 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231000+0000", 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3290000000000001e-06 2024-09-13T11:33:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231017+0000", 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6242000000000001e-05 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231025+0000", 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231094+0000", 2024-09-13T11:33:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9554999999999994e-05 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231171+0000", 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7082999999999994e-05 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236799+0000", 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056282260000000001 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237539+0000", 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073930700000000003 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237574+0000", 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5191999999999998e-05 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240013+0000", 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002439185 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240070+0000", 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6650999999999997e-05 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240109+0000", 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9131999999999997e-05 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:287 4.2 4:4297cb20:::smithi20438004-38:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.439679714, 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013317569, 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 287 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234931+0000", 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.192e-06 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234936+0000", 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0640000000000001e-06 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234953+0000", 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7292e-05 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234961+0000", 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235026+0000", 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5073000000000005e-05 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235111+0000", 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4635999999999994e-05 2024-09-13T11:33:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242149+0000", 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070384990000000001 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069074700000000002 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242936+0000", 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5730000000000007e-05 2024-09-13T11:33:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248138+0000", 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052018330000000003 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248197+0000", 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9512000000000002e-05 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248247+0000", 2024-09-13T11:33:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:310 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.425753346, 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095515259999999994, 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 310 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248857+0000", 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-06 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248863+0000", 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6529999999999999e-06 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248874+0000", 2024-09-13T11:33:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.093e-05 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248887+0000", 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3009e-05 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248932+0000", 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5445999999999998e-05 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249025+0000", 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3098999999999998e-05 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251503+0000", 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024772629999999999 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252148+0000", 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064507900000000001 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252433+0000", 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028559100000000002 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258304+0000", 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058703100000000001 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258361+0000", 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7658999999999998e-05 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258407+0000", 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6319999999999997e-05 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:313 4.2 4:4ba9f189:::smithi20438004-14:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.424874639000002, 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013958247999999999, 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 313 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249736+0000", 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9999999999999996e-07 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249741+0000", 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5929999999999996e-06 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249750+0000", 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7369999999999995e-06 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249757+0000", 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5360000000000004e-06 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249806+0000", 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9018999999999999e-05 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249882+0000", 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5943000000000001e-05 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258629+0000", 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087464589999999998 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259299+0000", 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067041799999999999 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259523+0000", 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223964 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263636+0000", 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041130539999999997 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263669+0000", 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2849999999999999e-05 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263693+0000", 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3774999999999999e-05 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1437 4.2 4:4b088bf8:::smithi20438004-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.870634252999999, 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014642921, 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1437 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803976+0000", 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.35e-06 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803984+0000", 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2080000000000003e-06 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803995+0000", 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804001+0000", 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2870000000000001e-06 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804096+0000", 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5424e-05 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804165+0000", 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8596000000000005e-05 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.818618+0000", 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452906999999999 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.429 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:24.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:24.437 INFO:tasks.rados.rados.0.smithi204.stdout:1514: finishing write tid 1 to smithi20438004-22 2024-09-13T11:33:24.437 INFO:tasks.rados.rados.0.smithi204.stdout:1514: finishing write tid 2 to smithi20438004-22 2024-09-13T11:33:24.437 INFO:tasks.rados.rados.0.smithi204.stdout:1514: finishing write tid 3 to smithi20438004-22 2024-09-13T11:33:24.437 INFO:tasks.rados.rados.0.smithi204.stdout:1514: finishing write tid 4 to smithi20438004-22 2024-09-13T11:33:24.437 INFO:tasks.rados.rados.0.smithi204.stdout:1514: finishing write tid 5 to smithi20438004-22 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1514: oid 22 updating version 0 to 92 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1514: oid 22 updating version 92 to 94 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1514: oid 22 version 94 is already newer than 91 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1514: oid 22 version 94 is already newer than 93 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 94 (ObjNum 305 snap 0 seq_num 305) dirty exists 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1514: left oid 22 (ObjNum 305 snap 0 seq_num 305) 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1510: done (5 left) 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1511: done (4 left) 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1512: done (3 left) 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1513: done (2 left) 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1514: done (1 left) 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1515: done (0 left) 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1516: write (excl) oid 1 current snap is 0 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1516: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-13T11:33:24.438 INFO:tasks.rados.rados.0.smithi204.stdout:1516: writing smithi20438004-1 from 759749 to 1315312 tid 1 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1516: writing smithi20438004-1 from 1765266 to 2557692 tid 2 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1516: writing smithi20438004-1 from 2964741 to 3159749 tid 3 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1517: read oid 25 snap -1 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1517: expect deleted 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1518: write oid 6 current snap is 0 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1516: finishing write tid 1 to smithi20438004-1 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1518: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1518: writing smithi20438004-6 from 776556 to 1407368 tid 1 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1518: writing smithi20438004-6 from 1823748 to 2476185 tid 2 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1518: writing smithi20438004-6 from 3108517 to 3176556 tid 3 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1519: write oid 24 current snap is 0 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1519: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1519: writing smithi20438004-24 from 793363 to 1499424 tid 1 2024-09-13T11:33:24.439 INFO:tasks.rados.rados.0.smithi204.stdout:1519: writing smithi20438004-24 from 2282230 to 3078325 tid 2 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1520: tier_flush oid 7 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1520: tier_flush oid 7 snap -1 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1516: finishing write tid 2 to smithi20438004-1 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1516: finishing write tid 3 to smithi20438004-1 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1517: done (4 left) 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1521: set_chunk oid 29 target oid 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1521: set_chunk oid 29 offset: 1103872 length: 5120 target oid a820312978ddb4c20c0f9b7fd4b429797b2e91fa74f61a055459629185aa8f45 offset: 0 snap -1 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1522: tier_flush oid 20 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1522: tier_flush oid 20 snap -1 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1516: finishing write tid 4 to smithi20438004-1 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1516: finishing write tid 5 to smithi20438004-1 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1516: oid 1 updating version 0 to 76 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1516: oid 1 updating version 76 to 78 2024-09-13T11:33:24.440 INFO:tasks.rados.rados.0.smithi204.stdout:1516: oid 1 version 78 is already newer than 77 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1516: oid 1 version 78 is already newer than 75 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 78 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1516: left oid 1 (ObjNum 307 snap 0 seq_num 307) 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1518: finishing write tid 1 to smithi20438004-6 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1516: done (5 left) 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1523: read oid 32 snap -1 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1523: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1524: write oid 22 current snap is 0 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1518: finishing write tid 2 to smithi20438004-6 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1518: finishing write tid 3 to smithi20438004-6 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1518: finishing write tid 4 to smithi20438004-6 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1518: finishing write tid 5 to smithi20438004-6 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1518: oid 6 updating version 0 to 81 2024-09-13T11:33:24.441 INFO:tasks.rados.rados.0.smithi204.stdout:1518: oid 6 version 81 is already newer than 79 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1518: oid 6 updating version 81 to 82 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1518: oid 6 version 82 is already newer than 80 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 82 (ObjNum 308 snap 0 seq_num 308) dirty exists 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1518: left oid 6 (ObjNum 308 snap 0 seq_num 308) 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: finishing write tid 1 to smithi20438004-24 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: finishing write tid 2 to smithi20438004-24 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: finishing write tid 3 to smithi20438004-24 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: finishing write tid 4 to smithi20438004-24 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: oid 24 updating version 0 to 109 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: oid 24 version 109 is already newer than 108 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: oid 24 updating version 109 to 110 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 110 (ObjNum 309 snap 0 seq_num 309) dirty exists 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1519: left oid 24 (ObjNum 309 snap 0 seq_num 309) 2024-09-13T11:33:24.442 INFO:tasks.rados.rados.0.smithi204.stdout:1524: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1524: writing smithi20438004-22 from 410170 to 1191480 tid 1 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1524: writing smithi20438004-22 from 1940712 to 2596818 tid 2 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1524: writing smithi20438004-22 from 3163363 to 3210170 tid 3 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1518: done (6 left) 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1519: done (5 left) 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1525: tier_flush oid 27 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1525: tier_flush oid 27 snap -1 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1521: finishing set_chunk to oid 29 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 29 set_chunk returned error code -95 offset: 1103872 length: 5120 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1523: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 294875 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1524: finishing write tid 1 to smithi20438004-22 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:1520: got (0) Success 2024-09-13T11:33:24.443 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1520: done (5 left) 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1521: done (4 left) 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1523: done (3 left) 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1526: read oid 25 snap -1 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1526: expect deleted 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1527: tier_flush oid 33 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1527: tier_flush oid 33 snap -1 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1524: finishing write tid 2 to smithi20438004-22 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1524: finishing write tid 3 to smithi20438004-22 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1522: got (0) Success 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 183 snap 0 seq_num 183) dirty exists 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1524: finishing write tid 4 to smithi20438004-22 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1524: finishing write tid 5 to smithi20438004-22 2024-09-13T11:33:24.444 INFO:tasks.rados.rados.0.smithi204.stdout:1524: oid 22 updating version 0 to 96 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1524: oid 22 updating version 96 to 98 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1524: oid 22 version 98 is already newer than 97 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1524: oid 22 version 98 is already newer than 95 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 98 (ObjNum 310 snap 0 seq_num 310) dirty exists 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1524: left oid 22 (ObjNum 310 snap 0 seq_num 310) 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1525: got (0) Success 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 0 (ObjNum 302 snap 0 seq_num 302) dirty exists 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1522: done (4 left) 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1524: done (3 left) 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1525: done (2 left) 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1528: read oid 3 snap -1 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1528: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1529: read oid 22 snap -1 2024-09-13T11:33:24.445 INFO:tasks.rados.rados.0.smithi204.stdout:1529: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1530: write (excl) oid 34 current snap is 0 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1530: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1530: writing smithi20438004-34 from 426977 to 883536 tid 1 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1530: writing smithi20438004-34 from 1482841 to 2282605 tid 2 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1530: writing smithi20438004-34 from 2958080 to 3226977 tid 3 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1531: read oid 11 snap -1 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1531: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1532: read oid 14 snap -1 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1532: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1533: read oid 36 snap -1 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1533: expect (ObjNum 299 snap 0 seq_num 299) 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1534: read oid 37 snap -1 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1534: expect deleted 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1535: read oid 16 snap -1 2024-09-13T11:33:24.446 INFO:tasks.rados.rados.0.smithi204.stdout:1535: expect (ObjNum 304 snap 0 seq_num 304) 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1536: read oid 46 snap -1 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1536: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1537: tier_evict oid 41 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1537: tier_evict oid 41 snap -1 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1538: read oid 5 snap -1 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1538: expect (ObjNum 286 snap 0 seq_num 286) 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1539: read oid 50 snap -1 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1539: expect deleted 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1527: got (2) No such file or directory 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1527: got expected ENOENT (src dne) 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1530: finishing write tid 1 to smithi20438004-34 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:1528: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:24.447 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1089205 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1529: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1530: finishing write tid 2 to smithi20438004-34 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1537: got (22) Invalid argument 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1526: done (13 left) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1527: done (12 left) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1528: done (11 left) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1529: done (10 left) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1534: done (9 left) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1537: done (8 left) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1540: read oid 42 snap -1 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1540: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1541: delete oid 33 current snap is 0 2024-09-13T11:33:24.448 INFO:tasks.rados.rados.0.smithi204.stdout:1532: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-13T11:33:24.449 INFO:tasks.rados.rados.0.smithi204.stdout:1535: expect (ObjNum 304 snap 0 seq_num 304) 2024-09-13T11:33:24.449 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:24.449 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:24.321+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:24.450 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:24.431+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: finishing write tid 3 to smithi20438004-34 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: finishing write tid 4 to smithi20438004-34 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: finishing write tid 5 to smithi20438004-34 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: oid 34 updating version 0 to 126 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: oid 34 updating version 126 to 128 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: oid 34 updating version 128 to 129 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: oid 34 version 129 is already newer than 127 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 129 (ObjNum 311 snap 0 seq_num 311) dirty exists 2024-09-13T11:33:24.450 INFO:tasks.rados.rados.0.smithi204.stdout:1530: left oid 34 (ObjNum 311 snap 0 seq_num 311) 2024-09-13T11:33:24.451 INFO:tasks.rados.rados.0.smithi204.stdout:1531: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:24.533 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:24.532+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:24.538 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:24.538 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:24.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:24.594 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:24.635 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:24.634+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:24.653 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:24.654 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:24.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:24.680 INFO:tasks.rados.rados.0.smithi204.stdout:1536: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:24.735 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:24.734+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:24.766 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 80.391839660000002, 2024-09-13T11:33:24.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27446322899999998, 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:24.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1100000000000005e-07 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372244+0000", 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.66e-06 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372256+0000", 2024-09-13T11:33:24.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1817000000000001e-05 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372264+0000", 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0220000000000004e-06 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372320+0000", 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6437999999999997e-05 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372329+0000", 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644121+0000", 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27179224899999999 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644271+0000", 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014990000000000001 2024-09-13T11:33:24.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644421+0000", 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015035600000000001 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646446+0000", 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00202454 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646524+0000", 2024-09-13T11:33:24.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8839999999999997e-05 2024-09-13T11:33:24.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646598+0000", 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3862999999999994e-05 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646654+0000", 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5609999999999998e-05 2024-09-13T11:33:24.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646701+0000", 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6844000000000001e-05 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:207 3.2 3:59a396eb:::smithi20438004-8 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.776 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.679442612999999, 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063519737000000007, 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 207 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7099999999999998e-07 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084636+0000", 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.559e-06 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084640+0000", 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.118e-06 2024-09-13T11:33:24.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084644+0000", 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8640000000000004e-06 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084687+0000", 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2295000000000001e-05 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084780+0000", 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2880000000000005e-05 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.106902+0000", 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022122822 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112195+0000", 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052927 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.121161+0000", 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089661740000000004 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148052+0000", 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026890595 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148105+0000", 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2673000000000002e-05 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148154+0000", 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9786000000000001e-05 2024-09-13T11:33:24.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:208 3.2 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.679423896999999, 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049409478, 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.784 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 208 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:24.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084654+0000", 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084657+0000", 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1769999999999998e-06 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084663+0000", 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8329999999999999e-06 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084668+0000", 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6200000000000004e-06 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084884+0000", 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021599499999999999 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.085046+0000", 2024-09-13T11:33:24.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016192600000000001 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.094613+0000", 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095668200000000002 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.101900+0000", 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072866160000000001 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.110249+0000", 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008349034 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133903+0000", 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023654272 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133976+0000", 2024-09-13T11:33:24.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2581000000000002e-05 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134063+0000", 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7169000000000005e-05 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.791 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:211 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.679382867999998, 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047621726000000003, 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 211 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.792 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084695+0000", 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6100000000000002e-07 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084697+0000", 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2419999999999999e-06 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8160000000000001e-06 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084706+0000", 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.313e-06 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086622+0000", 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001916212 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086720+0000", 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8077000000000003e-05 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100598+0000", 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013877195 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104958+0000", 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043607899999999998 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115608+0000", 2024-09-13T11:33:24.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010650138 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132197+0000", 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016589008999999998 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132257+0000", 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9948000000000003e-05 2024-09-13T11:33:24.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132316+0000", 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8625000000000002e-05 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:212 3.3 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.799 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.679365288, 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063179242999999996, 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 212 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7300000000000002e-07 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.255e-06 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084719+0000", 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4800000000000003e-06 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084724+0000", 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.527e-06 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086839+0000", 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021155240000000001 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086947+0000", 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107675 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109128+0000", 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022180947999999999 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.113055+0000", 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039269029999999998 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.119148+0000", 2024-09-13T11:33:24.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060933810000000001 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147786+0000", 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028638395000000001 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147838+0000", 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1405e-05 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147891+0000", 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3377000000000001e-05 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:216 3.2 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.673426040999999, 2024-09-13T11:33:24.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055588580999999998, 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 216 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090653+0000", 2024-09-13T11:33:24.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090663+0000", 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0708e-05 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090675+0000", 2024-09-13T11:33:24.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090688+0000", 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3193000000000001e-05 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096478+0000", 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057898150000000002 2024-09-13T11:33:24.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096694+0000", 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021582700000000001 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.103295+0000", 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006601048 2024-09-13T11:33:24.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108559+0000", 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052645449999999998 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.114944+0000", 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063845389999999998 2024-09-13T11:33:24.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146121+0000", 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031177449999999999 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146186+0000", 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5235999999999997e-05 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146240+0000", 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3368000000000001e-05 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:220 3.2 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.648250517000001, 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033838506999999997, 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 220 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115828+0000", 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115832+0000", 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.746e-06 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115842+0000", 2024-09-13T11:33:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2010000000000005e-06 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115847+0000", 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9740000000000001e-06 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115897+0000", 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0114999999999997e-05 2024-09-13T11:33:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116103+0000", 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020598799999999999 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135329+0000", 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019226422999999999 2024-09-13T11:33:24.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141061+0000", 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057316709999999998 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.142490+0000", 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001428805 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149552+0000", 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070625610000000002 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149602+0000", 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9826e-05 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149665+0000", 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3168000000000002e-05 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:226 3.3 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.629699070000001, 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.062274747999999998, 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 226 2024-09-13T11:33:24.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134379+0000", 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1850000000000001e-06 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134392+0000", 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2649999999999999e-05 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134404+0000", 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1615e-05 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134411+0000", 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3509999999999996e-06 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134474+0000", 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2673000000000001e-05 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134662+0000", 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018810100000000001 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149047+0000", 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014384902 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.154919+0000", 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058721379999999998 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.171898+0000", 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016979300999999999 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196539+0000", 2024-09-13T11:33:24.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024641252999999998 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196614+0000", 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5160999999999994e-05 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196653+0000", 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8417999999999997e-05 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:228 3.3 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.623416538000001, 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038699092999999997, 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.831 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:24.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140662+0000", 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140670+0000", 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8630000000000008e-06 2024-09-13T11:33:24.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140696+0000", 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5927000000000001e-05 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140708+0000", 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140804+0000", 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6137999999999993e-05 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140995+0000", 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191316 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.145999+0000", 2024-09-13T11:33:24.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050033129999999997 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152237+0000", 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062383320000000001 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160262+0000", 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080251520000000007 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179164+0000", 2024-09-13T11:33:24.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018901823000000002 2024-09-13T11:33:24.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179296+0000", 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013243400000000001 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179360+0000", 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3328000000000001e-05 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:229 3.3 3:d73fa370:::smithi20438004-30 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.623282510999999, 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057728689, 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.839 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 229 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140796+0000", 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9599999999999998e-07 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140800+0000", 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140809+0000", 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3669999999999997e-06 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140814+0000", 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:33:24.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141203+0000", 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038959299999999998 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141375+0000", 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017155499999999999 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.156365+0000", 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014989954999999999 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162876+0000", 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065109360000000002 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.187965+0000", 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025088847000000001 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198445+0000", 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010480171 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198487+0000", 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2635000000000001e-05 2024-09-13T11:33:24.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198523+0000", 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5941e-05 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:238 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:24.846 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.611317536000001, 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.059471032, 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 238 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.847 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152761+0000", 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152772+0000", 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1122e-05 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152782+0000", 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5179999999999993e-06 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152790+0000", 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7260000000000002e-06 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152856+0000", 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6304999999999996e-05 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153180+0000", 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000323617 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167201+0000", 2024-09-13T11:33:24.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014021264 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.172985+0000", 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057843349999999998 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188702+0000", 2024-09-13T11:33:24.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015716765000000001 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212106+0000", 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023403714999999999 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212169+0000", 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3224000000000003e-05 2024-09-13T11:33:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212231+0000", 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1841999999999995e-05 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.854 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:240 3.2 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.603827559999999, 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034186595, 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.855 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160251+0000", 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.562e-06 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160258+0000", 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6640000000000001e-06 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160269+0000", 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.152e-05 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160277+0000", 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4939999999999997e-06 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160327+0000", 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0268000000000001e-05 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160497+0000", 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017018900000000001 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.164999+0000", 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045015929999999999 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168431+0000", 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003431698 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.184172+0000", 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015741852000000001 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194334+0000", 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010161646999999999 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194390+0000", 2024-09-13T11:33:24.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6218999999999998e-05 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194436+0000", 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5889000000000001e-05 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:242 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 18] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:24.862 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.595439753000001, 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053252359999999999, 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 242 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.863 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168640+0000", 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168652+0000", 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1909000000000001e-05 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168668+0000", 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5888000000000001e-05 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168680+0000", 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.154e-05 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169643+0000", 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096305100000000003 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000139834 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174850+0000", 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005067841 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.178949+0000", 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040991209999999998 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193069+0000", 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014119166000000001 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221801+0000", 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028732905999999999 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221852+0000", 2024-09-13T11:33:24.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0958000000000002e-05 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221890+0000", 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7398999999999999e-05 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.870 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:244 3.2 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.583499729000003, 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040499786000000003, 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244 2024-09-13T11:33:24.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:24.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180579+0000", 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.499e-06 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180585+0000", 2024-09-13T11:33:24.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4649999999999999e-06 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180598+0000", 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180604+0000", 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5300000000000004e-06 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180669+0000", 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4859999999999999e-05 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180840+0000", 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171791 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.190029+0000", 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091888999999999998 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193312+0000", 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829579999999999 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.207773+0000", 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014461039 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.220989+0000", 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013215644 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221038+0000", 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9421e-05 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221077+0000", 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8899999999999997e-05 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:247 3.2 3:4f19c0b8:::smithi20438004-48 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.578262825000003, 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040924556000000001, 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.879 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 247 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:24.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185816+0000", 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9690000000000001e-06 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185825+0000", 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6980000000000004e-06 2024-09-13T11:33:24.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185842+0000", 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6492000000000001e-05 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185852+0000", 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0412e-05 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185951+0000", 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9312000000000001e-05 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186223+0000", 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027177000000000001 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208403+0000", 2024-09-13T11:33:24.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022179780999999999 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029604150000000001 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.223543+0000", 2024-09-13T11:33:24.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012179840000000001 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226659+0000", 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031159989999999999 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226698+0000", 2024-09-13T11:33:24.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9020999999999998e-05 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226739+0000", 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0846999999999997e-05 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.886 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.887 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:248 3.2 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:24.887 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:24.887 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.577939743999998, 2024-09-13T11:33:24.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035109013000000001, 2024-09-13T11:33:24.888 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.888 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.888 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:24.888 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:24.888 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 248 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186139+0000", 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.333e-06 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186157+0000", 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8488999999999999e-05 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186170+0000", 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2737e-05 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186181+0000", 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1279e-05 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186356+0000", 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174988 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186493+0000", 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137045 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.203390+0000", 2024-09-13T11:33:24.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016897169 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.206673+0000", 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829510000000001 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.214284+0000", 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076100830000000001 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221141+0000", 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068573740000000003 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221198+0000", 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6916999999999997e-05 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221246+0000", 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8647999999999999e-05 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.10.22:30 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.972902668000003, 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87173941200000005, 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:24.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791176+0000", 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791183+0000", 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.956e-06 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791193+0000", 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8649999999999996e-06 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791200+0000", 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.148e-06 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791263+0000", 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3038000000000002e-05 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791370+0000", 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010721999999999999 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791704+0000", 2024-09-13T11:33:24.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033406100000000003 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.792618+0000", 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091326999999999999 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.793087+0000", 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046985199999999999 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662893+0000", 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86980523300000001 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662914+0000", 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1319e-05 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.1.22:76 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:24.900 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.959255155000001, 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85814986199999999, 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804824+0000", 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5889999999999999e-06 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804830+0000", 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7139999999999996e-06 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804845+0000", 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5158e-05 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804853+0000", 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9349999999999994e-06 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804910+0000", 2024-09-13T11:33:24.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6501e-05 2024-09-13T11:33:24.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805017+0000", 2024-09-13T11:33:24.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107237 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805213+0000", 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019558100000000001 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806246+0000", 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001033225 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806260+0000", 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4072e-05 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662951+0000", 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85669131700000001 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662972+0000", 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0533000000000001e-05 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.906 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:9 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.918731454000003, 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81776348899999995, 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845348+0000", 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4090000000000001e-06 2024-09-13T11:33:24.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845360+0000", 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1420000000000001e-05 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845389+0000", 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9453000000000001e-05 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845404+0000", 2024-09-13T11:33:24.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4919000000000001e-05 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845526+0000", 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121588 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845617+0000", 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1119000000000006e-05 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845789+0000", 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017257000000000001 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846396+0000", 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060634700000000003 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846771+0000", 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037541099999999997 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663066+0000", 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81629477399999995 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663109+0000", 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3479000000000002e-05 2024-09-13T11:33:24.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:10 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.914995025000003, 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81407563599999999, 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:24.913 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849083+0000", 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6300000000000004e-07 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849089+0000", 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849101+0000", 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-05 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849106+0000", 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7309999999999998e-06 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849154+0000", 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8077000000000001e-05 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849235+0000", 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0770000000000001e-05 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849399+0000", 2024-09-13T11:33:24.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163385 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850247+0000", 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084889399999999999 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850266+0000", 2024-09-13T11:33:24.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8698e-05 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663146+0000", 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81287977300000003 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663158+0000", 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1928e-05 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.919 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.920 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:24.920 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:24.920 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:24.921 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:24.921 INFO:tasks.rados.rados.0.smithi204.stdout:1533: expect (ObjNum 299 snap 0 seq_num 299) 2024-09-13T11:33:24.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:24.922 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:24.920+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:25.021 INFO:tasks.rados.rados.0.smithi204.stdout:1538: expect (ObjNum 286 snap 0 seq_num 286) 2024-09-13T11:33:25.021 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:25.021+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:25.038 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:25.038 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:25.038 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:25.039 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:25.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:25.071 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1325662 2024-09-13T11:33:25.097 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:25.122 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:25.121+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:25.138 INFO:tasks.rados.rados.0.smithi204.stdout:1540: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-13T11:33:25.152 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:25.152 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:25.161 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:25.224 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:25.223+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:25.238 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2252836 2024-09-13T11:33:25.258 INFO:tasks.rados.rados.0.smithi204.stdout:1530: done (9 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1531: done (8 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1532: done (7 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1533: done (6 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1535: done (5 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1536: done (4 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1538: done (3 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1539: done (2 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1540: done (1 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1541: done (0 left) 2024-09-13T11:33:25.259 INFO:tasks.rados.rados.0.smithi204.stdout:1542: set_chunk oid 4 target oid 2024-09-13T11:33:25.261 INFO:tasks.rados.rados.0.smithi204.stdout:1542: set_chunk oid 4 offset: 441344 length: 52736 target oid e8213ebd0f980b5faae435a6cac169c33673298e8ac9b60156b9f2547bc7c084 offset: 0 snap -1 2024-09-13T11:33:25.261 INFO:tasks.rados.rados.0.smithi204.stdout:1543: read oid 14 snap -1 2024-09-13T11:33:25.261 INFO:tasks.rados.rados.0.smithi204.stdout:1543: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-13T11:33:25.261 INFO:tasks.rados.rados.0.smithi204.stdout:1544: read oid 7 snap -1 2024-09-13T11:33:25.261 INFO:tasks.rados.rados.0.smithi204.stdout:1544: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-13T11:33:25.261 INFO:tasks.rados.rados.0.smithi204.stdout:1545: write (excl) oid 3 current snap is 0 2024-09-13T11:33:25.261 INFO:tasks.rados.rados.0.smithi204.stdout:1545: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-13T11:33:25.267 INFO:tasks.rados.rados.0.smithi204.stdout:1545: writing smithi20438004-3 from 460591 to 1184001 tid 1 2024-09-13T11:33:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:25.270 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:25.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:25.270 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:25.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.270 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:201 3.1 3:b265d015:::smithi20438004-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.183044043000002, 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030631999, 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 201 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084693+0000", 2024-09-13T11:33:25.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4739999999999999e-06 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.884e-06 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2858e-05 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084748+0000", 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3113e-05 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084807+0000", 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8523999999999997e-05 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084930+0000", 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012317099999999999 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097750+0000", 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012820575000000001 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107462+0000", 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097120990000000001 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114842+0000", 2024-09-13T11:33:25.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073796520000000004 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115192+0000", 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035007400000000002 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115249+0000", 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7277000000000001e-05 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115323+0000", 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3298000000000002e-05 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:202 3.1 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.182962596000003, 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055756029999999998, 2024-09-13T11:33:25.277 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084773+0000", 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084777+0000", 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3540000000000002e-06 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084784+0000", 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084789+0000", 2024-09-13T11:33:25.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5480000000000001e-06 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085081+0000", 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000291673 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085197+0000", 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011550800000000001 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109819+0000", 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024622326999999999 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115554+0000", 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057346100000000002 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.139296+0000", 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023742025999999999 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140477+0000", 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011813660000000001 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140507+0000", 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0307e-05 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140528+0000", 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0934e-05 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.283 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:203 3.1 3:970ba768:::smithi20438004-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.182919828000003, 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055588820999999997, 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.284 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1900000000000003e-07 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084824+0000", 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3110000000000006e-06 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084831+0000", 2024-09-13T11:33:25.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4869999999999996e-06 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084839+0000", 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7840000000000004e-06 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085318+0000", 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047915700000000002 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085404+0000", 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5576999999999997e-05 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.108042+0000", 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022638703999999999 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115550+0000", 2024-09-13T11:33:25.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075072150000000002 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132897+0000", 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017347621000000001 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140260+0000", 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073629139999999999 2024-09-13T11:33:25.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140331+0000", 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0994999999999994e-05 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140404+0000", 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2537e-05 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:206 3.1 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.182884207000001, 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022595217000000001, 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:25.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1799999999999995e-07 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084854+0000", 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1650000000000002e-06 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084861+0000", 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1300000000000003e-06 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084868+0000", 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.55e-06 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085510+0000", 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064168099999999996 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085595+0000", 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5643999999999995e-05 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.091806+0000", 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062107220000000001 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107236+0000", 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015429897999999999 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107281+0000", 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4994000000000003e-05 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107302+0000", 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1163e-05 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107375+0000", 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2915000000000002e-05 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107446+0000", 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0837e-05 2024-09-13T11:33:25.296 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:210 3.1 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.182857071999997, 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047980961000000003, 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.297 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 210 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:25.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084883+0000", 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084890+0000", 2024-09-13T11:33:25.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0650000000000001e-06 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084896+0000", 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9000000000000003e-06 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085703+0000", 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000807097 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085822+0000", 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119098 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.104833+0000", 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019011389 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115543+0000", 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01070981 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127458+0000", 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011915189 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132784+0000", 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053251139999999997 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132825+0000", 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1329999999999999e-05 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132859+0000", 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:215 3.1 3:84f0de12:::smithi20438004-16 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.303 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.182830754999998, 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034172806999999999, 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 215 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0400000000000002e-07 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084907+0000", 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.666e-06 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084915+0000", 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9330000000000001e-06 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084920+0000", 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8289999999999998e-06 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085942+0000", 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001022029 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.086010+0000", 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8007000000000001e-05 2024-09-13T11:33:25.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.100909+0000", 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014898991 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107467+0000", 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065583270000000001 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.117910+0000", 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010443309 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.118971+0000", 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010605510000000001 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119023+0000", 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2114000000000003e-05 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119077+0000", 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3647000000000003e-05 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:218 3.1 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.157148206999999, 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036660768000000003, 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.310 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 218 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110588+0000", 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.761e-06 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110603+0000", 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4542000000000001e-05 2024-09-13T11:33:25.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110616+0000", 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2795e-05 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110628+0000", 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1959e-05 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114882+0000", 2024-09-13T11:33:25.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042543379999999999 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000143676 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.120883+0000", 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058572549999999996 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.123587+0000", 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027042709999999998 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144002+0000", 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020414505999999999 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147169+0000", 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031675470000000002 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147204+0000", 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4523999999999997e-05 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147247+0000", 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3594000000000003e-05 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:219 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.156683850999997, 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.054051957999999997, 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 219 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111052+0000", 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.093e-06 2024-09-13T11:33:25.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111059+0000", 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9410000000000001e-06 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111069+0000", 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0519999999999999e-05 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111077+0000", 2024-09-13T11:33:25.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8290000000000008e-06 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115373+0000", 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042952490000000001 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115459+0000", 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6367999999999997e-05 2024-09-13T11:33:25.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.129610+0000", 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014151285 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.135388+0000", 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057775270000000002 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149276+0000", 2024-09-13T11:33:25.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013888773 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165015+0000", 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015738682 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165065+0000", 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0170000000000002e-05 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165103+0000", 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7521000000000002e-05 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:221 3.1 3:baf08732:::smithi20438004-22 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.15129108, 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034943881000000003, 2024-09-13T11:33:25.323 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 221 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:25.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116445+0000", 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1680000000000001e-06 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116451+0000", 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0379999999999996e-06 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116461+0000", 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0604e-05 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116469+0000", 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8359999999999992e-06 2024-09-13T11:33:25.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116516+0000", 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6332999999999998e-05 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116579+0000", 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3825999999999995e-05 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127789+0000", 2024-09-13T11:33:25.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011209869000000001 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132875+0000", 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050855780000000003 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149082+0000", 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016207213000000002 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151312+0000", 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002230358 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151344+0000", 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1081000000000003e-05 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151388+0000", 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3977000000000003e-05 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:230 3.1 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.121140607000001, 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046965185999999999, 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-13T11:33:25.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146596+0000", 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.717e-06 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146606+0000", 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146620+0000", 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4343e-05 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146630+0000", 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3989999999999993e-06 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146705+0000", 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5135999999999993e-05 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146828+0000", 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122714 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165695+0000", 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018866951 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.171485+0000", 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057905220000000002 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185089+0000", 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013604132999999999 2024-09-13T11:33:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193461+0000", 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083717840000000002 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193507+0000", 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5652000000000003e-05 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193559+0000", 2024-09-13T11:33:25.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2624000000000003e-05 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:231 3.1 3:ab075b6a:::smithi20438004-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.120931255999999, 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033558088, 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.336 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 231 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146804+0000", 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3300000000000002e-07 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146808+0000", 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4790000000000001e-06 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146814+0000", 2024-09-13T11:33:25.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3230000000000003e-06 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146817+0000", 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4769999999999999e-06 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146955+0000", 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013720299999999999 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147052+0000", 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7231000000000005e-05 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.156769+0000", 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097173290000000002 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165122+0000", 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083524589999999996 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.173409+0000", 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082871379999999994 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180256+0000", 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006846871 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180302+0000", 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6545999999999998e-05 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180362+0000", 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9499000000000001e-05 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:235 3.1 3:be4a0686:::smithi20438004-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.117932830000001, 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035212407000000001, 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.342 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 235 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149803+0000", 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0580000000000001e-06 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149810+0000", 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:25.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149820+0000", 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0464e-05 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149826+0000", 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149863+0000", 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7416999999999998e-05 2024-09-13T11:33:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149983+0000", 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011927800000000001 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.160853+0000", 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010870216 2024-09-13T11:33:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.167925+0000", 2024-09-13T11:33:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070717159999999996 2024-09-13T11:33:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180224+0000", 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012298972 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184892+0000", 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046685839999999999 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184959+0000", 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7105000000000002e-05 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185014+0000", 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5099000000000002e-05 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:236 3.1 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.117818915000001, 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059325392999999997, 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 236 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6599999999999996e-07 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149920+0000", 2024-09-13T11:33:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149929+0000", 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3769999999999996e-06 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149937+0000", 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150123+0000", 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018567200000000001 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150254+0000", 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130426 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181070+0000", 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030816856 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.190793+0000", 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097220870000000008 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197985+0000", 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007192431 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209174+0000", 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011189536999999999 2024-09-13T11:33:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209209+0000", 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4359000000000001e-05 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209241+0000", 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2369000000000001e-05 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:237 3.1 3:9a2730a7:::smithi20438004-38 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.115866128, 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052738543999999998, 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 237 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151870+0000", 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:33:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151875+0000", 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2859999999999999e-06 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151890+0000", 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5557999999999999e-05 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151898+0000", 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7500000000000003e-06 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151934+0000", 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5988000000000002e-05 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151999+0000", 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4394999999999996e-05 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.177811+0000", 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025812831000000001 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185336+0000", 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075243209999999996 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197387+0000", 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012051345999999999 2024-09-13T11:33:25.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204491+0000", 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071034999999999996 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204551+0000", 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0003000000000001e-05 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204607+0000", 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:245 3.1 3:adf25eb4:::smithi20438004-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.086677080000001, 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.03651658, 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 245 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181059+0000", 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181066+0000", 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5119999999999997e-06 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181078+0000", 2024-09-13T11:33:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2333e-05 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181085+0000", 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4449999999999999e-06 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185114+0000", 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040285579999999998 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1935000000000007e-05 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.195343+0000", 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010137147000000001 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200849+0000", 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005505874 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.203835+0000", 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002985705 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217494+0000", 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013658926 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217538+0000", 2024-09-13T11:33:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4048999999999998e-05 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217574+0000", 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6618000000000001e-05 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:249 3.1 3:81faf980:::smithi20438004-50 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.073758431000002, 2024-09-13T11:33:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038339094999999997, 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 249 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193977+0000", 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.11e-06 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193984+0000", 2024-09-13T11:33:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0794e-05 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194002+0000", 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.374e-06 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194061+0000", 2024-09-13T11:33:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9485999999999999e-05 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194291+0000", 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000229321 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222511+0000", 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028219767999999999 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.227779+0000", 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052686319999999997 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.231941+0000", 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041617809999999998 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232226+0000", 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028545799999999999 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232277+0000", 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0445000000000002e-05 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232315+0000", 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584999999999997e-05 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:44 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 37.999334703000002, 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019379767999999999, 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268402+0000", 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268410+0000", 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2069999999999995e-06 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268426+0000", 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5588999999999999e-05 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268437+0000", 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0773e-05 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270340+0000", 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001903231 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270350+0000", 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0543000000000001e-05 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278988+0000", 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086371350000000006 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279079+0000", 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1012000000000002e-05 2024-09-13T11:33:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279189+0000", 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110598 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281564+0000", 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023743929999999998 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281584+0000", 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0655000000000001e-05 2024-09-13T11:33:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287627+0000", 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060427750000000002 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287716+0000", 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9470999999999998e-05 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287780+0000", 2024-09-13T11:33:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3329000000000003e-05 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:45 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 37.999276934999997, 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019566733999999999, 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6100000000000001e-07 2024-09-13T11:33:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268462+0000", 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6430000000000001e-06 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268469+0000", 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268475+0000", 2024-09-13T11:33:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8039999999999998e-06 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270391+0000", 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001916713 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270397+0000", 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8490000000000002e-06 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280010+0000", 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096124780000000007 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280095+0000", 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5506000000000004e-05 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280235+0000", 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000139405 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281596+0000", 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001361615 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281676+0000", 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0259999999999994e-05 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287969+0000", 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062924000000000001 2024-09-13T11:33:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287999+0000", 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0270999999999999e-05 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288025+0000", 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5488999999999999e-05 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:26 3.1 3:a6c449f0:::2fa16cfaf91e48313b717fb6e16c839aad29b3524900af3b2b4692c59d034ccb:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 37.999051565999999, 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019443548000000001, 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7500000000000001e-07 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268686+0000", 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1270000000000001e-06 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268689+0000", 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8709999999999999e-06 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268694+0000", 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4530000000000004e-06 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270498+0000", 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00180423 2024-09-13T11:33:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270502+0000", 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.625e-06 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280882+0000", 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010380263000000001 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280973+0000", 2024-09-13T11:33:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1199000000000006e-05 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281088+0000", 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011427800000000001 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282421+0000", 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013333590000000001 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282432+0000", 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.113e-05 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288074+0000", 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056417469999999999 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288104+0000", 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0402000000000001e-05 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288127+0000", 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2589e-05 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:28 3.1 3:96eef999:::4534b522f789f408dc72053dbc198d7d54ba3a77b9806bd26ef033725ea1a210:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 37.999035825999997, 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019504840999999998, 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6300000000000001e-07 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268702+0000", 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.441e-06 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268709+0000", 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.888e-06 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268714+0000", 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.519e-06 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270524+0000", 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018093740000000001 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270527+0000", 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.484e-06 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281702+0000", 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011174584 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281828+0000", 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012597699999999999 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281975+0000", 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014744000000000001 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282783+0000", 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080754199999999996 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282862+0000", 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9326999999999995e-05 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288155+0000", 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005293056 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288183+0000", 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8152000000000001e-05 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288204+0000", 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0693999999999999e-05 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.400 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:25.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:33:25.403 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:25.401+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:25.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:25.405 INFO:tasks.rados.rados.0.smithi204.stdout:1545: writing smithi20438004-3 from 1716158 to 2119591 tid 2 2024-09-13T11:33:25.409 INFO:tasks.rados.rados.0.smithi204.stdout:1545: writing smithi20438004-3 from 2729279 to 3169419 tid 3 2024-09-13T11:33:25.409 INFO:tasks.rados.rados.0.smithi204.stdout:1546: write (excl) oid 33 current snap is 0 2024-09-13T11:33:25.409 INFO:tasks.rados.rados.0.smithi204.stdout:1542: finishing set_chunk to oid 4 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 441344 length: 52736 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 0 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1546: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1546: writing smithi20438004-33 from 477398 to 1276057 tid 1 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1546: writing smithi20438004-33 from 1774640 to 2438084 tid 2 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1546: writing smithi20438004-33 from 2873055 to 3277398 tid 3 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1542: done (4 left) 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1547: read oid 17 snap -1 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1547: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1548: read oid 42 snap -1 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1548: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1549: delete oid 38 current snap is 0 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:1544: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-13T11:33:25.410 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 91658 2024-09-13T11:33:25.429 INFO:tasks.rados.rados.0.smithi204.stdout:1543: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-13T11:33:25.450 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2403544 2024-09-13T11:33:25.494 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 919503 2024-09-13T11:33:25.502 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:25.501+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:25.513 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:25.514 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:25.514 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:25.514 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:25.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:25.523 INFO:tasks.rados.rados.0.smithi204.stdout:1545: finishing write tid 1 to smithi20438004-3 2024-09-13T11:33:25.523 INFO:tasks.rados.rados.0.smithi204.stdout:1545: finishing write tid 2 to smithi20438004-3 2024-09-13T11:33:25.523 INFO:tasks.rados.rados.0.smithi204.stdout:1545: finishing write tid 3 to smithi20438004-3 2024-09-13T11:33:25.523 INFO:tasks.rados.rados.0.smithi204.stdout:1545: finishing write tid 4 to smithi20438004-3 2024-09-13T11:33:25.523 INFO:tasks.rados.rados.0.smithi204.stdout:1545: finishing write tid 5 to smithi20438004-3 2024-09-13T11:33:25.523 INFO:tasks.rados.rados.0.smithi204.stdout:1545: oid 3 updating version 0 to 43 2024-09-13T11:33:25.523 INFO:tasks.rados.rados.0.smithi204.stdout:1545: oid 3 updating version 43 to 44 2024-09-13T11:33:25.524 INFO:tasks.rados.rados.0.smithi204.stdout:1545: oid 3 updating version 44 to 45 2024-09-13T11:33:25.524 INFO:tasks.rados.rados.0.smithi204.stdout:1545: oid 3 version 45 is already newer than 42 2024-09-13T11:33:25.524 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 45 (ObjNum 313 snap 0 seq_num 313) dirty exists 2024-09-13T11:33:25.524 INFO:tasks.rados.rados.0.smithi204.stdout:1545: left oid 3 (ObjNum 313 snap 0 seq_num 313) 2024-09-13T11:33:25.524 INFO:tasks.rados.rados.0.smithi204.stdout:1546: finishing write tid 1 to smithi20438004-33 2024-09-13T11:33:25.524 INFO:tasks.rados.rados.0.smithi204.stdout:1546: finishing write tid 2 to smithi20438004-33 2024-09-13T11:33:25.524 INFO:tasks.rados.rados.0.smithi204.stdout:1547: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:25.603 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:25.602+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:25.626 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:25.626 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:25.626 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:25.626 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:25.626 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:25.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1546: finishing write tid 3 to smithi20438004-33 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1546: finishing write tid 4 to smithi20438004-33 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1543: done (6 left) 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1544: done (5 left) 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1545: done (4 left) 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1547: done (3 left) 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1549: done (2 left) 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1550: read oid 4 snap -1 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1550: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-13T11:33:25.684 INFO:tasks.rados.rados.0.smithi204.stdout:1551: set_chunk oid 16 target oid 2024-09-13T11:33:25.689 INFO:tasks.rados.rados.0.smithi204.stdout:1551: set_chunk oid 16 offset: 448512 length: 694784 target oid c27f4031b445a6a6643b6efd32ce47ee7eb14c4deac67a8e1bb1ff6cebc01111 offset: 0 snap -1 2024-09-13T11:33:25.689 INFO:tasks.rados.rados.0.smithi204.stdout:1552: read oid 50 snap -1 2024-09-13T11:33:25.689 INFO:tasks.rados.rados.0.smithi204.stdout:1552: expect deleted 2024-09-13T11:33:25.689 INFO:tasks.rados.rados.0.smithi204.stdout:1553: set_chunk oid 1 target oid 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1553: set_chunk oid 1 offset: 894976 length: 1478656 target oid ae2c28a6802641e2106eb84a7b3f62a978a2525f3e5d0a3466712563b1729296 offset: 0 snap -1 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1554: read oid 40 snap -1 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1554: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1555: write oid 47 current snap is 0 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1546: finishing write tid 5 to smithi20438004-33 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1546: oid 33 updating version 0 to 89 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1546: oid 33 updating version 89 to 92 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1546: oid 33 version 92 is already newer than 91 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1546: oid 33 version 92 is already newer than 90 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 92 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1546: left oid 33 (ObjNum 314 snap 0 seq_num 314) 2024-09-13T11:33:25.701 INFO:tasks.rados.rados.0.smithi204.stdout:1548: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-13T11:33:25.704 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:25.703+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:254 4.0 4:04aeeab9:::smithi20438004-5:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.537754051, 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022108323999999999, 2024-09-13T11:33:25.742 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 254 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201594+0000", 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201603+0000", 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.568e-06 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201619+0000", 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6540000000000001e-05 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201629+0000", 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3200000000000006e-06 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201685+0000", 2024-09-13T11:33:25.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6078000000000001e-05 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201747+0000", 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2242999999999993e-05 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:25.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220103+0000", 2024-09-13T11:33:25.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01835548 2024-09-13T11:33:25.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220937+0000", 2024-09-13T11:33:25.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083429299999999995 2024-09-13T11:33:25.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221938+0000", 2024-09-13T11:33:25.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010010710000000001 2024-09-13T11:33:25.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223567+0000", 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001628846 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223638+0000", 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1165000000000001e-05 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223701+0000", 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3120999999999994e-05 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.752 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:285 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:25.752 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:25.752 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.504769592000002, 2024-09-13T11:33:25.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012191677, 2024-09-13T11:33:25.752 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 285 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:25.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234578+0000", 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234583+0000", 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9110000000000001e-06 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234593+0000", 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234598+0000", 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9509999999999997e-06 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234639+0000", 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1826999999999998e-05 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234688+0000", 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9039000000000003e-05 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239203+0000", 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045143739999999998 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239979+0000", 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077664100000000001 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.242634+0000", 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026544899999999998 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246683+0000", 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040491499999999996 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246723+0000", 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0429000000000002e-05 2024-09-13T11:33:25.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246769+0000", 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5430999999999999e-05 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1715 4.f 4:fea8c5b2:::smithi20438004-39:head [write 769787~704463,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.470960652000002, 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020210576000000001, 2024-09-13T11:33:25.759 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1715 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:25.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268388+0000", 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269172+0000", 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078478 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269184+0000", 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.182e-05 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269194+0000", 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8570000000000007e-06 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269269+0000", 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4795000000000005e-05 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269283+0000", 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275426+0000", 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061428050000000003 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275540+0000", 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011432199999999999 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275658+0000", 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118071 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278252+0000", 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594224 2024-09-13T11:33:25.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279142+0000", 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088964600000000001 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288321+0000", 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091785600000000005 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288363+0000", 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2957000000000002e-05 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288492+0000", 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012896899999999999 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288597+0000", 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010429000000000001 2024-09-13T11:33:25.766 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.767 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.767 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.767 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1716 4.f 4:fea8c5b2:::smithi20438004-39:head [write 2267961~101826] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:25.767 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:25.767 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.470133795000002, 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019610079999999998, 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1716 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269214+0000", 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9699999999999995e-07 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269305+0000", 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1330999999999997e-05 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269314+0000", 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9209999999999998e-06 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269321+0000", 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269363+0000", 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2388999999999997e-05 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269372+0000", 2024-09-13T11:33:25.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:25.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277175+0000", 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078032580000000004 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277247+0000", 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1773999999999994e-05 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277354+0000", 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010704000000000001 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288308+0000", 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010953353000000001 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288328+0000", 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0353e-05 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288640+0000", 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031178999999999999 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288706+0000", 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6749999999999996e-05 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288765+0000", 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8876999999999999e-05 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288823+0000", 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7806999999999997e-05 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1717 4.f 4:fea8c5b2:::smithi20438004-39:head [setxattr _header (56) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.470009533000002, 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019744788999999999, 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.777 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1717 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:25.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269338+0000", 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269342+0000", 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.924e-06 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269349+0000", 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9619999999999996e-06 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269355+0000", 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1210000000000003e-06 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269399+0000", 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4360000000000002e-05 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5570000000000001e-06 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278293+0000", 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088867549999999997 2024-09-13T11:33:25.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278372+0000", 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9561999999999995e-05 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278492+0000", 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119654 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288335+0000", 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098426840000000008 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288672+0000", 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033785899999999998 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288863+0000", 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019105599999999999 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288904+0000", 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0738000000000002e-05 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288949+0000", 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4443000000000001e-05 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289082+0000", 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013346199999999999 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1718 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.469980059000001, 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019946973, 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1718 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:25.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:25.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8999999999999996e-07 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269373+0000", 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8629999999999996e-06 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269379+0000", 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4979999999999997e-06 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269383+0000", 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7260000000000002e-06 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269443+0000", 2024-09-13T11:33:25.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9616e-05 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269451+0000", 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279055+0000", 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009604586 2024-09-13T11:33:25.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279104+0000", 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8131999999999999e-05 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289119+0000", 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010015616 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289177+0000", 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8060999999999999e-05 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289314+0000", 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000136525 2024-09-13T11:33:25.793 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1732 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.469955564000003, 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017069699000000001, 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.794 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1732 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269392+0000", 2024-09-13T11:33:25.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-13T11:33:25.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269395+0000", 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-06 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269400+0000", 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.198e-06 2024-09-13T11:33:25.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9410000000000004e-06 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269514+0000", 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108124 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269524+0000", 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279170+0000", 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096457209999999995 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279248+0000", 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8147999999999998e-05 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286461+0000", 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072129789999999996 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1733 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.469912588, 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0187283, 2024-09-13T11:33:25.801 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1733 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269435+0000", 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7800000000000001e-07 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269438+0000", 2024-09-13T11:33:25.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.535e-06 2024-09-13T11:33:25.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269447+0000", 2024-09-13T11:33:25.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6559999999999998e-06 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269452+0000", 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.972e-06 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269552+0000", 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9820000000000003e-05 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269556+0000", 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286496+0000", 2024-09-13T11:33:25.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016940182000000002 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286550+0000", 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3640999999999997e-05 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288163+0000", 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016126720000000001 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1734 4.f 4:f653a832:::smithi20438004-19:head [read 1042153~2571319,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.469846687, 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021098286000000001, 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1734 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269501+0000", 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269504+0000", 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.58e-06 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269511+0000", 2024-09-13T11:33:25.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6200000000000001e-06 2024-09-13T11:33:25.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269517+0000", 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7030000000000003e-06 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269580+0000", 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269583+0000", 2024-09-13T11:33:25.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5020000000000001e-06 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289341+0000", 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019757429 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289385+0000", 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3939000000000001e-05 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.290598+0000", 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012136130000000001 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.12.0:167 4.6 e51/31 4:6978b14d:::smithi20438004-23:head v 51'52, mlcod=45'50)", 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 38.547732344000003, 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013003298, 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.814 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191616+0000", 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0270000000000001e-06 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196708+0000", 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050925720000000001 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196728+0000", 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9616999999999999e-05 2024-09-13T11:33:25.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196738+0000", 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8330000000000006e-06 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196817+0000", 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9225000000000001e-05 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196841+0000", 2024-09-13T11:33:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3972e-05 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204573+0000", 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077323629999999999 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204618+0000", 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4688999999999999e-05 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.818 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1770 4.0 4:001d90c2:::smithi20438004-28:head [read 0~2218524,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 38.469557725999998, 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011933050000000001, 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1770 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.819 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269791+0000", 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269795+0000", 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269803+0000", 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269810+0000", 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.246e-06 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273037+0000", 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032276700000000002 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273048+0000", 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0511e-05 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.277997+0000", 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049497539999999998 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278150+0000", 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152406 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281722+0000", 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00357224 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.823 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1771 4.0 4:001d90c2:::smithi20438004-28:head [read 633380~1585144,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 38.469527722000002, 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013022562, 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.824 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1771 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269820+0000", 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4000000000000002e-07 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269821+0000", 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9269999999999999e-06 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269825+0000", 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8789999999999996e-06 2024-09-13T11:33:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269829+0000", 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4740000000000001e-06 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273144+0000", 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033155459999999999 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273155+0000", 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.062e-05 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282280+0000", 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091252620000000003 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282403+0000", 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012240499999999999 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282842+0000", 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043900900000000002 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1772 4.0 4:001d90c2:::smithi20438004-28:head [sparse-read 2038646~179878] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 38.469514277000002, 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013177762000000001, 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.829 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1772 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9499999999999998e-07 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269834+0000", 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269837+0000", 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.005e-06 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269843+0000", 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6389999999999997e-06 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273188+0000", 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033454840000000001 2024-09-13T11:33:25.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273196+0000", 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0830000000000004e-06 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282874+0000", 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096776810000000005 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282924+0000", 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0095000000000001e-05 2024-09-13T11:33:25.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.283010+0000", 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6216000000000003e-05 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2324 4.16 4:6978b14d:::smithi20438004-23:head [write 544593~799729,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.602266296, 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14427186, 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.834 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2324 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7100000000000001e-07 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137660+0000", 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057879000000000003 2024-09-13T11:33:25.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137671+0000", 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0518e-05 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137675+0000", 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137754+0000", 2024-09-13T11:33:25.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8492000000000007e-05 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137765+0000", 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1027e-05 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273433+0000", 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13566771999999999 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273569+0000", 2024-09-13T11:33:25.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013616999999999999 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273699+0000", 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130142 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281227+0000", 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075284699999999998 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281259+0000", 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1610999999999999e-05 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281298+0000", 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9138999999999999e-05 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281352+0000", 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.840 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2325 4.16 4:6978b14d:::smithi20438004-23:head [write 1761799~776065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.59560098, 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13997515799999999, 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.841 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2325 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143748+0000", 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000001e-06 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144385+0000", 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000637125 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144396+0000", 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0633e-05 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144401+0000", 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2469999999999999e-06 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144457+0000", 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5566999999999998e-05 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144467+0000", 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8819999999999996e-06 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275371+0000", 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13090408000000001 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275440+0000", 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8749000000000002e-05 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275530+0000", 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0236999999999997e-05 2024-09-13T11:33:25.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282215+0000", 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066849240000000001 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283575+0000", 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001360775 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283629+0000", 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283721+0000", 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1799999999999995e-05 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2326 4.16 4:6978b14d:::smithi20438004-23:head [write 2948352~396241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:25.847 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.591235559000001, 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13583619799999999, 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2326 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:25.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148113+0000", 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148305+0000", 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000192008 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148316+0000", 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0293e-05 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148327+0000", 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1489e-05 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148398+0000", 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0421999999999998e-05 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148407+0000", 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1090000000000004e-06 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279356+0000", 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13094884700000001 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279444+0000", 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279534+0000", 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9573000000000005e-05 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282581+0000", 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030473990000000001 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283856+0000", 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001275307 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283901+0000", 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5238999999999998e-05 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283947+0000", 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5972e-05 2024-09-13T11:33:25.853 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2327 4.16 4:6978b14d:::smithi20438004-23:head [setxattr _header (56) in=7b,truncate 3344593] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.591008059, 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13580746099999999, 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.854 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2327 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.34e-07 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148341+0000", 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0999999999999998e-06 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148363+0000", 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2045999999999999e-05 2024-09-13T11:33:25.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148369+0000", 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.665e-06 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148463+0000", 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4475999999999995e-05 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148474+0000", 2024-09-13T11:33:25.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0319999999999999e-05 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280931+0000", 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132457142 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280994+0000", 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2693000000000004e-05 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:25.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281088+0000", 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3993000000000006e-05 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282632+0000", 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001544199 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283989+0000", 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013571670000000001 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284048+0000", 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9070999999999997e-05 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284146+0000", 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8255e-05 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2328 4.16 4:6978b14d:::smithi20438004-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:25.860 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.590973544000001, 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13592306100000001, 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2328 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148374+0000", 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6899999999999999e-07 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148375+0000", 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148378+0000", 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0989999999999999e-06 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148381+0000", 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148500+0000", 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119052 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148509+0000", 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8759999999999994e-06 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281377+0000", 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132868244 2024-09-13T11:33:25.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281422+0000", 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4909e-05 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284171+0000", 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027497300000000001 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284216+0000", 2024-09-13T11:33:25.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4626999999999998e-05 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284296+0000", 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0275e-05 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2687 4.10 4:0c9026a3:::smithi20438004-36:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.667521359999999, 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029783952999999998, 2024-09-13T11:33:25.866 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2687 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071827+0000", 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.096e-06 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071831+0000", 2024-09-13T11:33:25.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8210000000000001e-06 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071843+0000", 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1715e-05 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071848+0000", 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071899+0000", 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0290000000000001e-05 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071988+0000", 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.101609+0000", 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0296212 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3509 4.f 4:f5aacf61:::smithi20438004-27:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 1.9600021829999998, 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022491955000000001, 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:25.871 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3509 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779347+0000", 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9209999999999999e-06 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779354+0000", 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7980000000000006e-06 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779373+0000", 2024-09-13T11:33:25.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289999999999999e-05 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779399+0000", 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779457+0000", 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8888000000000003e-05 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:25.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779523+0000", 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5995999999999996e-05 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.801837+0000", 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022313116000000001 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:25.875 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:25.876 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:25.876 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:25.876 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:25.876 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-09-13T11:33:25.876 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T11:33:25.876 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T11:33:25.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok config set filestore_inject_stall 3 2024-09-13T11:33:25.878 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:25.876+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:25.878 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:25.879 INFO:tasks.rados.rados.0.smithi204.stdout:1550: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-13T11:33:25.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:25.977 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:25.976+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:25.980 INFO:tasks.rados.rados.0.smithi204.stdout:1551: finishing set_chunk to oid 16 2024-09-13T11:33:25.981 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 448512 length: 694784 2024-09-13T11:33:25.981 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 0 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-09-13T11:33:25.981 INFO:tasks.rados.rados.0.smithi204.stdout:1553: finishing set_chunk to oid 1 2024-09-13T11:33:25.981 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 78 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-09-13T11:33:25.981 INFO:tasks.rados.rados.0.smithi204.stdout:1554: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-13T11:33:25.981 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 335192 2024-09-13T11:33:25.984 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:25.984 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:25.984 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:25.984 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:25.990 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:25.990 INFO:teuthology.orchestra.run.smithi118.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T11:33:25.990 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:25.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:26.078 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:26.077+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:26.079 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 297174 2024-09-13T11:33:26.127 INFO:tasks.rados.rados.0.smithi204.stdout:1555: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-09-13T11:33:26.132 INFO:tasks.rados.rados.0.smithi204.stdout:1555: writing smithi20438004-47 from 511012 to 1060169 tid 1 2024-09-13T11:33:26.137 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:26.137 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:26.137 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:26.137 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:26.137 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:26.140 INFO:tasks.rados.rados.0.smithi204.stdout:1555: writing smithi20438004-47 from 1775251 to 2326011 tid 2 2024-09-13T11:33:26.145 INFO:tasks.rados.rados.0.smithi204.stdout:1555: writing smithi20438004-47 from 3095195 to 3311012 tid 3 2024-09-13T11:33:26.145 INFO:tasks.rados.rados.0.smithi204.stdout:1546: done (7 left) 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1548: done (6 left) 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1550: done (5 left) 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1551: done (4 left) 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1552: done (3 left) 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1553: done (2 left) 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1554: done (1 left) 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1556: write (excl) oid 3 current snap is 0 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1555: finishing write tid 1 to smithi20438004-47 2024-09-13T11:33:26.146 INFO:tasks.rados.rados.0.smithi204.stdout:1556: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-13T11:33:26.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:26.151 INFO:tasks.rados.rados.0.smithi204.stdout:1556: writing smithi20438004-3 from 527819 to 1152225 tid 1 2024-09-13T11:33:26.157 INFO:tasks.rados.rados.0.smithi204.stdout:1556: writing smithi20438004-3 from 1833733 to 2244504 tid 2 2024-09-13T11:33:26.163 INFO:tasks.rados.rados.0.smithi204.stdout:1556: writing smithi20438004-3 from 2722618 to 3324199 tid 3 2024-09-13T11:33:26.163 INFO:tasks.rados.rados.0.smithi204.stdout:1557: write (excl) oid 16 current snap is 0 2024-09-13T11:33:26.163 INFO:tasks.rados.rados.0.smithi204.stdout:1557: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-13T11:33:26.169 INFO:tasks.rados.rados.0.smithi204.stdout:1557: writing smithi20438004-16 from 544626 to 1244281 tid 1 2024-09-13T11:33:26.176 INFO:tasks.rados.rados.0.smithi204.stdout:1557: writing smithi20438004-16 from 1892215 to 2446644 tid 2 2024-09-13T11:33:26.180 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:26.180+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1557: writing smithi20438004-16 from 3150041 to 3344626 tid 3 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1558: read oid 25 snap -1 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1558: expect deleted 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1559: write oid 38 current snap is 0 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1555: finishing write tid 2 to smithi20438004-47 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1555: finishing write tid 3 to smithi20438004-47 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1555: finishing write tid 4 to smithi20438004-47 2024-09-13T11:33:26.182 INFO:tasks.rados.rados.0.smithi204.stdout:1559: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-13T11:33:26.189 INFO:tasks.rados.rados.0.smithi204.stdout:1559: writing smithi20438004-38 from 561433 to 1336337 tid 1 2024-09-13T11:33:26.194 INFO:tasks.rados.rados.0.smithi204.stdout:1559: writing smithi20438004-38 from 1950697 to 2365137 tid 2 2024-09-13T11:33:26.200 INFO:tasks.rados.rados.0.smithi204.stdout:1559: writing smithi20438004-38 from 2777464 to 3361433 tid 3 2024-09-13T11:33:26.200 INFO:tasks.rados.rados.0.smithi204.stdout:1560: set_chunk oid 22 target oid 2024-09-13T11:33:26.209 INFO:tasks.rados.rados.0.smithi204.stdout:1560: set_chunk oid 22 offset: 986112 length: 1071616 target oid daa2548987192ebea26e4d0f985e0ac31e32052b28fb077c0392e1315ed018f6 offset: 0 snap -1 2024-09-13T11:33:26.209 INFO:tasks.rados.rados.0.smithi204.stdout:1561: write oid 40 current snap is 0 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:1555: finishing write tid 5 to smithi20438004-47 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:1555: oid 47 updating version 0 to 43 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:1555: oid 47 updating version 43 to 45 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:1555: oid 47 version 45 is already newer than 44 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:1555: oid 47 updating version 45 to 46 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 46 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:1555: left oid 47 (ObjNum 316 snap 0 seq_num 316) 2024-09-13T11:33:26.210 INFO:tasks.rados.rados.0.smithi204.stdout:1561: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-13T11:33:26.215 INFO:tasks.rados.rados.0.smithi204.stdout:1561: writing smithi20438004-40 from 578240 to 1144746 tid 1 2024-09-13T11:33:26.221 INFO:tasks.rados.rados.0.smithi204.stdout:1561: writing smithi20438004-40 from 1609179 to 2167277 tid 2 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1561: writing smithi20438004-40 from 2804887 to 3356637 tid 3 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1555: done (6 left) 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1562: delete oid 42 current snap is 0 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1556: finishing write tid 1 to smithi20438004-3 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1556: finishing write tid 2 to smithi20438004-3 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1556: finishing write tid 3 to smithi20438004-3 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1556: finishing write tid 4 to smithi20438004-3 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1556: finishing write tid 5 to smithi20438004-3 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1556: oid 3 updating version 0 to 49 2024-09-13T11:33:26.229 INFO:tasks.rados.rados.0.smithi204.stdout:1556: oid 3 version 49 is already newer than 47 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1556: oid 3 version 49 is already newer than 48 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1556: oid 3 version 49 is already newer than 46 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 49 (ObjNum 317 snap 0 seq_num 317) dirty exists 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1556: left oid 3 (ObjNum 317 snap 0 seq_num 317) 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: finishing write tid 1 to smithi20438004-16 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: finishing write tid 2 to smithi20438004-16 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: finishing write tid 3 to smithi20438004-16 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: finishing write tid 4 to smithi20438004-16 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: finishing write tid 5 to smithi20438004-16 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: oid 16 updating version 0 to 68 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: oid 16 version 68 is already newer than 66 2024-09-13T11:33:26.230 INFO:tasks.rados.rados.0.smithi204.stdout:1557: oid 16 version 68 is already newer than 67 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1557: oid 16 version 68 is already newer than 65 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 68 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1557: left oid 16 (ObjNum 318 snap 0 seq_num 318) 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: oid 38 updating version 0 to 128 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: oid 38 version 128 is already newer than 127 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: oid 38 updating version 128 to 130 2024-09-13T11:33:26.231 INFO:tasks.rados.rados.0.smithi204.stdout:1559: oid 38 version 130 is already newer than 129 2024-09-13T11:33:26.232 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 130 (ObjNum 319 snap 0 seq_num 319) dirty exists 2024-09-13T11:33:26.232 INFO:tasks.rados.rados.0.smithi204.stdout:1559: left oid 38 (ObjNum 319 snap 0 seq_num 319) 2024-09-13T11:33:26.232 INFO:tasks.rados.rados.0.smithi204.stdout:1560: finishing set_chunk to oid 22 2024-09-13T11:33:26.232 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 98 (ObjNum 310 snap 0 seq_num 310) dirty exists 2024-09-13T11:33:26.232 INFO:tasks.rados.rados.0.smithi204.stdout:1561: finishing write tid 1 to smithi20438004-40 2024-09-13T11:33:26.232 INFO:tasks.rados.rados.0.smithi204.stdout:1561: finishing write tid 2 to smithi20438004-40 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1556: done (6 left) 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1557: done (5 left) 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1558: done (4 left) 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1559: done (3 left) 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1560: done (2 left) 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1562: done (1 left) 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1563: read oid 50 snap -1 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1563: expect deleted 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1564: read oid 13 snap -1 2024-09-13T11:33:26.233 INFO:tasks.rados.rados.0.smithi204.stdout:1564: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:26.234 INFO:tasks.rados.rados.0.smithi204.stdout:1565: set_chunk oid 17 target oid 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1565: set_chunk oid 17 offset: 728576 length: 478720 target oid 2f4c64522177438cfeb312201303c2bdb6e01ada8d1edd163bed2dafa84a56ab offset: 0 snap -1 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1566: read oid 16 snap -1 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1566: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1567: read oid 46 snap -1 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1567: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1568: write oid 29 current snap is 0 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1561: finishing write tid 3 to smithi20438004-40 2024-09-13T11:33:26.237 INFO:tasks.rados.rados.0.smithi204.stdout:1568: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-13T11:33:26.244 INFO:tasks.rados.rados.0.smithi204.stdout:1568: writing smithi20438004-29 from 611854 to 1328858 tid 1 2024-09-13T11:33:26.252 INFO:tasks.rados.rados.0.smithi204.stdout:1568: writing smithi20438004-29 from 2126143 to 2687910 tid 2 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:26.256 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.858974844999999, 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078075499999999999, 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:26.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394848+0000", 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397492+0000", 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026441139999999999 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397513+0000", 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0381e-05 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397520+0000", 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4789999999999999e-06 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397586+0000", 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5687999999999999e-05 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397612+0000", 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5870000000000001e-05 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402610+0000", 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049984199999999999 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402654+0000", 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4153000000000002e-05 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "pg_update_log_missing(4.e epoch 38/31 rep_tid 32 entries 38'23 (0'0) error 4:7022fc62:::smithi20438004-6:head by client.4631.0:787 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 38'20 roll_forward_to 38'20)", 2024-09-13T11:33:26.261 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.854247938, 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057797359999999997, 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:26.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399574+0000", 2024-09-13T11:33:26.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8500000000000001e-07 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399583+0000", 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4139999999999994e-06 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399602+0000", 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9176000000000001e-05 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399611+0000", 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2520000000000005e-06 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.399661+0000", 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9688000000000003e-05 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.405353+0000", 2024-09-13T11:33:26.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056925209999999999 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 49.222569753000002, 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081668139999999997, 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.266 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031253+0000", 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7900000000000007e-07 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037708+0000", 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064553029999999999 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037720+0000", 2024-09-13T11:33:26.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2062000000000001e-05 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037745+0000", 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4766999999999999e-05 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037797+0000", 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2198999999999999e-05 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037815+0000", 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8178000000000001e-05 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039386+0000", 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015708110000000001 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039419+0000", 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2515000000000003e-05 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.270 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1715 4.f e51/50 4:fea8c5b2:::smithi20438004-39:head v 51'67, mlcod=0'0)", 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.978052415000001, 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063741700000000002, 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:26.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275770+0000", 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9999999999999997e-07 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276160+0000", 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038996700000000002 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276171+0000", 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0702e-05 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276177+0000", 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7750000000000004e-06 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276230+0000", 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2751999999999999e-05 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276252+0000", 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2055000000000001e-05 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282092+0000", 2024-09-13T11:33:26.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058396180000000004 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282143+0000", 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1601000000000003e-05 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2285 4.e e57/51 4:7984cf84:::smithi20438004-46:head v 57'55, mlcod=56'53)", 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:26.275 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 31.022587239, 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055185929999999996, 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231234+0000", 2024-09-13T11:33:26.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.231235+0000", 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3799999999999997e-07 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232465+0000", 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001230182 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232475+0000", 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9650000000000003e-06 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232480+0000", 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1599999999999997e-06 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232538+0000", 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7711999999999999e-05 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.232552+0000", 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.363e-05 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236731+0000", 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.004179833 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:55.236753+0000", 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1472999999999999e-05 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.279 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4d4c8512:::smithi20438004-29:head, version: 55'82, data_included: [405504~978944,1437696~69632], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4d4c8512:::smithi20438004-29:head@55'82, size: 2806769, copy_subset: [(0, 2806769)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1507328, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 28.919676424999999, 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064952710000000004, 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:26.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334146+0000", 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9500000000000002e-07 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339863+0000", 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057170859999999997 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339880+0000", 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6705999999999999e-05 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339897+0000", 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7663999999999999e-05 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339963+0000", 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5808000000000004e-05 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339972+0000", 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6289999999999999e-06 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.340640+0000", 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066878300000000005 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2515 4.12 4:4d4c8512:::smithi20438004-29:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.175825190000001, 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008644934, 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.284 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2515 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:26.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077997+0000", 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5999999999999991e-07 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078584+0000", 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058694699999999999 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078595+0000", 2024-09-13T11:33:26.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0808999999999999e-05 2024-09-13T11:33:26.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078600+0000", 2024-09-13T11:33:26.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9779999999999996e-06 2024-09-13T11:33:26.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078667+0000", 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6445999999999994e-05 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078732+0000", 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5061000000000006e-05 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078820+0000", 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8208999999999995e-05 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080991+0000", 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021715419999999998 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084271+0000", 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032798079999999999 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086533+0000", 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022614810000000001 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086566+0000", 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2959000000000001e-05 2024-09-13T11:33:26.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086641+0000", 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5734000000000004e-05 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2516 4.12 4:4d4c8512:::smithi20438004-29:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.175210483000001, 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082031369999999992, 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2516 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.293 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078612+0000", 2024-09-13T11:33:26.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078631+0000", 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9823999999999999e-05 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078642+0000", 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.011e-05 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078647+0000", 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.079994+0000", 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001346835 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080073+0000", 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8721999999999996e-05 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080232+0000", 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000159698 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.081037+0000", 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080509799999999997 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084295+0000", 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032578360000000001 2024-09-13T11:33:26.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086714+0000", 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024189799999999998 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086749+0000", 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4331e-05 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086814+0000", 2024-09-13T11:33:26.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5703000000000004e-05 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2517 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.175166718, 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083158850000000003, 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2517 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9800000000000004e-07 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078658+0000", 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1940000000000001e-06 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078664+0000", 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1859999999999997e-06 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078668+0000", 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6820000000000002e-06 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080476+0000", 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018074160000000001 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080525+0000", 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9036000000000003e-05 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086839+0000", 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063138650000000001 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086884+0000", 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5717999999999998e-05 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086971+0000", 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6290000000000002e-05 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2537 4.12 4:4b088bf8:::smithi20438004-24:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.134871333, 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077763800000000003, 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.304 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2537 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118952+0000", 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0159999999999998e-06 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118976+0000", 2024-09-13T11:33:26.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4227999999999999e-05 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118987+0000", 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119009+0000", 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2232999999999999e-05 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119058+0000", 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8427999999999998e-05 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119146+0000", 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.126727+0000", 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075806049999999998 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.308 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2746 4.e e66/51 4:727deab8:::smithi20438004-48:head v 66'59, mlcod=57'57)", 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.066705529, 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056203759999999998, 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.187116+0000", 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188837+0000", 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001720435 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188852+0000", 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5741000000000001e-05 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188867+0000", 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4123e-05 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188930+0000", 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.368e-05 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.188949+0000", 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9004999999999998e-05 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.192684+0000", 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037346020000000001 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.192736+0000", 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2547999999999999e-05 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:2890 4.f e69/59 4:fea8c5b2:::smithi20438004-39:head v 69'88, mlcod=66'85)", 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:10.980484+0000", 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 15.273337793, 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054412949999999996, 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.313 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980484+0000", 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980484+0000", 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980485+0000", 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0100000000000001e-06 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980757+0000", 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027216700000000003 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980770+0000", 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3157e-05 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980778+0000", 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1030000000000002e-06 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980829+0000", 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1093e-05 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.980858+0000", 2024-09-13T11:33:26.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8861e-05 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.985876+0000", 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050179380000000004 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:10.985925+0000", 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8965999999999998e-05 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3033 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3663926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.755321478000001, 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083775140000000008, 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3033 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.318 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:26.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4499999999999998e-07 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498503+0000", 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4889999999999998e-06 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498508+0000", 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7079999999999996e-06 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498511+0000", 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1879999999999998e-06 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503489+0000", 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049776849999999999 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503558+0000", 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9128000000000006e-05 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.506878+0000", 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033199710000000001 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3042 4.12 4:4b088bf8:::smithi20438004-24:head [write 753013~645493] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.143480389, 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099953939999999995, 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.322 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3042 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110345+0000", 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.563e-06 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111084+0000", 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073883200000000003 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111101+0000", 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7039999999999999e-05 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111110+0000", 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9060000000000008e-06 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111178+0000", 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8503000000000005e-05 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111183+0000", 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2959999999999998e-06 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112893+0000", 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001709728 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112975+0000", 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1757999999999999e-05 2024-09-13T11:33:26.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.113113+0000", 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013859500000000001 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119968+0000", 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068545869999999997 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120007+0000", 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584000000000002e-05 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120085+0000", 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8226999999999995e-05 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120188+0000", 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103243 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120337+0000", 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014853199999999999 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3043 4.12 4:4b088bf8:::smithi20438004-24:head [write 2056248~296765] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.115337+0000", 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.138484142999999, 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055019099999999996, 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.329 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3043 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115337+0000", 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115337+0000", 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115339+0000", 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4270000000000001e-06 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115555+0000", 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000215837 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115570+0000", 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4893e-05 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115577+0000", 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.109e-06 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.117911+0000", 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023347099999999998 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.118019+0000", 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000107509 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.118199+0000", 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018026300000000001 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120021+0000", 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001821853 2024-09-13T11:33:26.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120381+0000", 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035969399999999998 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120668+0000", 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000287414 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120748+0000", 2024-09-13T11:33:26.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9374000000000003e-05 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120839+0000", 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1827e-05 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3044 4.12 4:4b088bf8:::smithi20438004-24:head [setxattr _header (56) in=7b,truncate 2353013] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.138229696, 2024-09-13T11:33:26.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057109630000000003, 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3044 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115592+0000", 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5400000000000001e-07 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115596+0000", 2024-09-13T11:33:26.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.002e-06 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115601+0000", 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5689999999999998e-06 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115605+0000", 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3900000000000003e-06 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119338+0000", 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037322150000000001 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119398+0000", 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0336e-05 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119499+0000", 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010123099999999999 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120393+0000", 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00089431999999999999 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120578+0000", 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000184914 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121182+0000", 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060329600000000002 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121216+0000", 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4062e-05 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121303+0000", 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7074e-05 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.341 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3045 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.138210840999999, 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060597120000000001, 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3045 2024-09-13T11:33:26.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.03e-07 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115613+0000", 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.598e-06 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115616+0000", 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7500000000000001e-06 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115619+0000", 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.089e-06 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119902+0000", 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042830250000000002 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119956+0000", 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3075999999999998e-05 2024-09-13T11:33:26.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121340+0000", 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013844669999999999 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121395+0000", 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4721e-05 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121670+0000", 2024-09-13T11:33:26.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027568299999999998 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3122 4.12 4:4b088bf8:::smithi20438004-24:head [write 521083~714336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:16.374254+0000", 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.8795672240000005, 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055176000000000001, 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.347 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3122 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.374254+0000", 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.374254+0000", 2024-09-13T11:33:26.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.374256+0000", 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.858e-06 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375083+0000", 2024-09-13T11:33:26.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00082709999999999999 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375098+0000", 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4369000000000001e-05 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375107+0000", 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9919999999999996e-06 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375175+0000", 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8802999999999998e-05 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375244+0000", 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8093e-05 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.375366+0000", 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121946 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.377750+0000", 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023846610000000002 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379180+0000", 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001430136 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379610+0000", 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000429667 2024-09-13T11:33:26.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379709+0000", 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9419999999999993e-05 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:16.379772+0000", 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2555e-05 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3369 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3663926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.2447225199999998, 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077891109999999996, 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3369 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009100+0000", 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1900000000000002e-07 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009105+0000", 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6899999999999997e-06 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009114+0000", 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1079999999999999e-06 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009121+0000", 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5320000000000003e-06 2024-09-13T11:33:26.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.010369+0000", 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001247915 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.010466+0000", 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7579999999999997e-05 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.016888+0000", 2024-09-13T11:33:26.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064216669999999998 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3370 4.12 4:4d4c8512:::smithi20438004-29:head [read 32972~3630954,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.2446892930000004, 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010949787000000001, 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:26.358 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3370 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:26.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009133+0000", 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7299999999999995e-07 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009136+0000", 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5090000000000002e-06 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:26.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009144+0000", 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2880000000000004e-06 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009150+0000", 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.016932+0000", 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077818109999999996 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.017048+0000", 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011644799999999999 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.020082+0000", 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030339849999999999 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:26.362 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:26.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:26.365 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:33:26.366 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:26.368 INFO:tasks.rados.rados.0.smithi204.stdout:1568: writing smithi20438004-29 from 3259733 to 3411854 tid 3 2024-09-13T11:33:26.368 INFO:tasks.rados.rados.0.smithi204.stdout:1569: tier_evict oid 14 2024-09-13T11:33:26.368 INFO:tasks.rados.rados.0.smithi204.stdout:1569: tier_evict oid 14 snap -1 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1570: write oid 45 current snap is 0 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: finishing write tid 4 to smithi20438004-40 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: finishing write tid 5 to smithi20438004-40 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: oid 40 updating version 0 to 100 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: oid 40 updating version 100 to 101 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: oid 40 updating version 101 to 102 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: oid 40 updating version 102 to 103 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 103 (ObjNum 320 snap 0 seq_num 320) dirty exists 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: left oid 40 (ObjNum 320 snap 0 seq_num 320) 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1570: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1570: writing smithi20438004-45 from 628661 to 1420914 tid 1 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1570: writing smithi20438004-45 from 2184625 to 2606403 tid 2 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1570: writing smithi20438004-45 from 3403509 to 3428661 tid 3 2024-09-13T11:33:26.369 INFO:tasks.rados.rados.0.smithi204.stdout:1561: done (8 left) 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1563: done (7 left) 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1571: read oid 8 snap -1 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1571: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1572: read oid 34 snap -1 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1572: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1573: set_chunk oid 1 target oid 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1573: set_chunk oid 1 offset: 222208 length: 2536448 target oid 431fb6bab5751654b8359ae618286a09a6bd9a92fdda9f40ed45a6fc781ea5e2 offset: 0 snap -1 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1574: write oid 36 current snap is 0 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1565: finishing set_chunk to oid 17 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk returned error code -95 offset: 728576 length: 478720 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 289 snap 0 seq_num 289) dirty exists 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1574: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1574: writing smithi20438004-36 from 645468 to 1112970 tid 1 2024-09-13T11:33:26.370 INFO:tasks.rados.rados.0.smithi204.stdout:1574: writing smithi20438004-36 from 1726754 to 2292190 tid 2 2024-09-13T11:33:26.371 INFO:tasks.rados.rados.0.smithi204.stdout:1574: writing smithi20438004-36 from 2798226 to 3395064 tid 3 2024-09-13T11:33:26.371 INFO:tasks.rados.rados.0.smithi204.stdout:1565: done (10 left) 2024-09-13T11:33:26.371 INFO:tasks.rados.rados.0.smithi204.stdout:1575: delete oid 31 current snap is 0 2024-09-13T11:33:26.371 INFO:tasks.rados.rados.0.smithi204.stdout:1566: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-13T11:33:26.371 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:26.362+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:26.464 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:26.463+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:26.465 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:26.473 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:26.473 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:26.473 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:26.473 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:26.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:26.533 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:26.533 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:26.549 INFO:tasks.rados.rados.0.smithi204.stdout:1567: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-13T11:33:26.551 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:26.551 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:26.551 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:26.565 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:26.564+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:26.568 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:26.568 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:26.584 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:26.584 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:26.585 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:26.585 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:26.595 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:26.601 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:26.601 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:26.618 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:26.619 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:26.635 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:26.636 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:26.653 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:26.653 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:26.666 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:26.665+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:26.669 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:26.670 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:26.687 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:26.688 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:26.698 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1487677 2024-09-13T11:33:26.699 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:26.699 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:26.704 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:26.705 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:26.705 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:26.708 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:26.722 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:26.722 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:26.736 INFO:tasks.rados.rados.0.smithi204.stdout:1564: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:26.736 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:26.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:26.739 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:26.757 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:33:26.756+0000 7f1bdce1b640 0 -- 172.21.15.38:0/1183077633 >> v2:172.21.15.204:6812/501019151 conn(0x7f1bd821fc70 msgr2=0x7f1bd8222060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:33:26.759 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:26.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:26.767 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:26.766+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:26.816 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:26.816 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:26.824 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:26.868 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:26.866+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:26.899 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 618102 2024-09-13T11:33:26.967 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:26.967 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:26.969 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:26.968+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:26.971 INFO:tasks.rados.rados.0.smithi204.stdout:1568: finishing write tid 1 to smithi20438004-29 2024-09-13T11:33:26.971 INFO:tasks.rados.rados.0.smithi204.stdout:1568: finishing write tid 2 to smithi20438004-29 2024-09-13T11:33:26.971 INFO:tasks.rados.rados.0.smithi204.stdout:1564: done (10 left) 2024-09-13T11:33:26.971 INFO:tasks.rados.rados.0.smithi204.stdout:1566: done (9 left) 2024-09-13T11:33:26.971 INFO:tasks.rados.rados.0.smithi204.stdout:1567: done (8 left) 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1575: done (7 left) 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1576: tier_promote oid 48 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1569: got (22) Invalid argument 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1568: finishing write tid 3 to smithi20438004-29 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1568: finishing write tid 4 to smithi20438004-29 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1569: done (7 left) 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1577: delete oid 26 current snap is 0 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1568: finishing write tid 5 to smithi20438004-29 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1568: oid 29 updating version 0 to 111 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1568: oid 29 updating version 111 to 113 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1568: oid 29 updating version 113 to 114 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:1568: oid 29 version 114 is already newer than 112 2024-09-13T11:33:26.972 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 114 (ObjNum 322 snap 0 seq_num 322) dirty exists 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1568: left oid 29 (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: finishing write tid 1 to smithi20438004-45 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: finishing write tid 2 to smithi20438004-45 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: finishing write tid 3 to smithi20438004-45 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: finishing write tid 4 to smithi20438004-45 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: finishing write tid 5 to smithi20438004-45 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: oid 45 updating version 0 to 75 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: oid 45 updating version 75 to 78 2024-09-13T11:33:26.973 INFO:tasks.rados.rados.0.smithi204.stdout:1570: oid 45 version 78 is already newer than 76 2024-09-13T11:33:26.974 INFO:tasks.rados.rados.0.smithi204.stdout:1570: oid 45 version 78 is already newer than 77 2024-09-13T11:33:26.974 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 78 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-09-13T11:33:26.974 INFO:tasks.rados.rados.0.smithi204.stdout:1570: left oid 45 (ObjNum 323 snap 0 seq_num 323) 2024-09-13T11:33:26.974 INFO:tasks.rados.rados.0.smithi204.stdout:1572: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-13T11:33:26.976 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:27.070 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:27.069+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:27.121 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:27.121 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:27.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:27.171 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:27.170+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:27.184 INFO:tasks.rados.rados.0.smithi204.stdout:1571: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:27.185 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 371424 2024-09-13T11:33:27.273 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:27.272+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:27.273 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:27.273 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:27.283 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:27.373 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:27.372+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:27.381 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1688272 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1573: finishing set_chunk to oid 1 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk returned error code -95 offset: 222208 length: 2536448 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1574: finishing write tid 1 to smithi20438004-36 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1568: done (7 left) 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1570: done (6 left) 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1571: done (5 left) 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1572: done (4 left) 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1573: done (3 left) 2024-09-13T11:33:27.427 INFO:tasks.rados.rados.0.smithi204.stdout:1577: done (2 left) 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1578: set_chunk oid 25 target oid 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1578: set_chunk oid 25 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1579: read oid 39 snap -1 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1579: expect (ObjNum 247 snap 0 seq_num 247) 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1574: finishing write tid 2 to smithi20438004-36 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1574: finishing write tid 3 to smithi20438004-36 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1580: read oid 9 snap -1 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1580: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1581: delete oid 6 current snap is 0 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1574: finishing write tid 4 to smithi20438004-36 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1574: finishing write tid 5 to smithi20438004-36 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1574: oid 36 updating version 0 to 67 2024-09-13T11:33:27.428 INFO:tasks.rados.rados.0.smithi204.stdout:1574: oid 36 version 67 is already newer than 66 2024-09-13T11:33:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:1574: oid 36 version 67 is already newer than 65 2024-09-13T11:33:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:1574: oid 36 updating version 67 to 68 2024-09-13T11:33:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 68 (ObjNum 324 snap 0 seq_num 324) dirty exists 2024-09-13T11:33:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:1574: left oid 36 (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:1576: got (0) Success 2024-09-13T11:33:27.429 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 69 (ObjNum 260 snap 0 seq_num 260) dirty exists 2024-09-13T11:33:27.429 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:27.429 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1578: finishing set_chunk to oid 25 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1578: got expected ENOENT (src dne) 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1574: done (5 left) 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1576: done (4 left) 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1578: done (3 left) 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1581: done (2 left) 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1582: set_chunk oid 50 target oid 2024-09-13T11:33:27.430 INFO:tasks.rados.rados.0.smithi204.stdout:1582: set_chunk oid 50 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1583: read oid 31 snap -1 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1583: expect deleted 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1584: tier_evict oid 44 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1584: tier_evict oid 44 snap -1 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1585: read oid 34 snap -1 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1585: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1586: read oid 15 snap -1 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1586: expect (ObjNum 301 snap 0 seq_num 301) 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1587: read oid 36 snap -1 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1587: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1588: tier_evict oid 46 2024-09-13T11:33:27.431 INFO:tasks.rados.rados.0.smithi204.stdout:1588: tier_evict oid 46 snap -1 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1589: read oid 43 snap -1 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1589: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1590: read oid 21 snap -1 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1590: expect deleted 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1591: read oid 6 snap -1 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1591: expect deleted 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1592: read oid 48 snap -1 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1592: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1593: read oid 27 snap -1 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1593: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1594: write oid 49 current snap is 0 2024-09-13T11:33:27.432 INFO:tasks.rados.rados.0.smithi204.stdout:1594: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-13T11:33:27.436 INFO:tasks.rados.rados.0.smithi204.stdout:1594: writing smithi20438004-49 from 712696 to 1197547 tid 1 2024-09-13T11:33:27.443 INFO:tasks.rados.rados.0.smithi204.stdout:1594: writing smithi20438004-49 from 1960682 to 2533456 tid 2 2024-09-13T11:33:27.448 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1594: writing smithi20438004-49 from 3307918 to 3512696 tid 3 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1595: read oid 33 snap -1 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1595: expect (ObjNum 314 snap 0 seq_num 314) 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1588: got (22) Invalid argument 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1588: done (15 left) 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1596: read oid 29 snap -1 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1596: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1582: finishing set_chunk to oid 50 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1582: got expected ENOENT (src dne) 2024-09-13T11:33:27.450 INFO:tasks.rados.rados.0.smithi204.stdout:1580: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-13T11:33:27.493 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:27.492+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:27.512 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 828571 2024-09-13T11:33:27.555 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:27.556 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:27.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:27.575 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:27.573+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:27.596 INFO:tasks.rados.rados.0.smithi204.stdout:1584: got (22) Invalid argument 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1580: done (15 left) 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1582: done (14 left) 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1583: done (13 left) 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1584: done (12 left) 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1590: done (11 left) 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1591: done (10 left) 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1597: write oid 3 current snap is 0 2024-09-13T11:33:27.597 INFO:tasks.rados.rados.0.smithi204.stdout:1597: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-13T11:33:27.603 INFO:tasks.rados.rados.0.smithi204.stdout:1597: writing smithi20438004-3 from 729503 to 1289603 tid 1 2024-09-13T11:33:27.611 INFO:tasks.rados.rados.0.smithi204.stdout:1597: writing smithi20438004-3 from 2019164 to 2735596 tid 2 2024-09-13T11:33:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:1597: writing smithi20438004-3 from 3335341 to 3529503 tid 3 2024-09-13T11:33:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:1598: write (excl) oid 24 current snap is 0 2024-09-13T11:33:27.616 INFO:tasks.rados.rados.0.smithi204.stdout:1598: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-13T11:33:27.623 INFO:tasks.rados.rados.0.smithi204.stdout:1598: writing smithi20438004-24 from 746310 to 1381659 tid 1 2024-09-13T11:33:27.630 INFO:tasks.rados.rados.0.smithi204.stdout:1598: writing smithi20438004-24 from 2077646 to 2654089 tid 2 2024-09-13T11:33:27.635 INFO:tasks.rados.rados.0.smithi204.stdout:1598: writing smithi20438004-24 from 3362764 to 3546310 tid 3 2024-09-13T11:33:27.635 INFO:tasks.rados.rados.0.smithi204.stdout:1599: tier_evict oid 45 2024-09-13T11:33:27.635 INFO:tasks.rados.rados.0.smithi204.stdout:1599: tier_evict oid 45 snap -1 2024-09-13T11:33:27.635 INFO:tasks.rados.rados.0.smithi204.stdout:1585: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-13T11:33:27.664 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2565967 2024-09-13T11:33:27.676 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:27.674+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:27.702 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:27.702 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:27.708 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1005752 2024-09-13T11:33:27.710 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:27.776 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:27.775+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:27.848 INFO:tasks.rados.rados.0.smithi204.stdout:1589: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:27.848 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:27.854 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:27.854 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:27.863 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:27.877 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:27.876+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:27.925 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:27.967 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:27.967 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:27.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:27.977 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:27.976+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:28.063 INFO:tasks.rados.rados.0.smithi204.stdout:1592: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:28.063 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:28.078 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:28.078+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:28.097 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:28.098 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:28.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:28.111 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:28.158 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 220277 2024-09-13T11:33:28.179 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:28.178+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:28.205 INFO:tasks.rados.rados.0.smithi204.stdout:1579: expect (ObjNum 247 snap 0 seq_num 247) 2024-09-13T11:33:28.237 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:28.256 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:28.256 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:28.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:28.270 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 138636 2024-09-13T11:33:28.280 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:28.279+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:28.302 INFO:tasks.rados.rados.0.smithi204.stdout:1586: expect (ObjNum 301 snap 0 seq_num 301) 2024-09-13T11:33:28.350 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:28.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:33:28.372 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:28.372 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:28.381 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:28.380+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:28.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:28.447 INFO:tasks.rados.rados.0.smithi204.stdout:1594: finishing write tid 1 to smithi20438004-49 2024-09-13T11:33:28.447 INFO:tasks.rados.rados.0.smithi204.stdout:1594: finishing write tid 2 to smithi20438004-49 2024-09-13T11:33:28.447 INFO:tasks.rados.rados.0.smithi204.stdout:1587: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:28.482 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:28.481+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:28.532 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:28.532 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:28.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:28.583 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:28.581+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:28.588 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 342246 2024-09-13T11:33:28.650 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:28.650 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:28.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:28.665 INFO:tasks.rados.rados.0.smithi204.stdout:1593: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-13T11:33:28.684 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1861679 2024-09-13T11:33:28.684 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:28.683+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:28.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:28.767 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:28.768 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:28.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:28.785 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:28.783+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:1594: finishing write tid 3 to smithi20438004-49 2024-09-13T11:33:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:1594: finishing write tid 4 to smithi20438004-49 2024-09-13T11:33:28.793 INFO:tasks.rados.rados.0.smithi204.stdout:1594: finishing write tid 5 to smithi20438004-49 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:1594: oid 49 updating version 0 to 34 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:1594: oid 49 updating version 34 to 37 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:1594: oid 49 version 37 is already newer than 35 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:1594: oid 49 version 37 is already newer than 36 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 37 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:1594: left oid 49 (ObjNum 328 snap 0 seq_num 328) 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:1596: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:28.794 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:28.886 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:28.884+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:28.892 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:28.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:28.910 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:28.911 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:28.911 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:28.925 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:28.925 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:28.927 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:28.927 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:28.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:28.945 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:28.945 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:28.963 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:28.964 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:28.964 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3035213 2024-09-13T11:33:28.973 INFO:tasks.rados.rados.0.smithi204.stdout:1595: expect (ObjNum 314 snap 0 seq_num 314) 2024-09-13T11:33:28.981 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:28.981 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:28.982 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2823615 2024-09-13T11:33:28.987 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:28.985+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:28.992 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:28.997 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:28.997 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:29.014 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:29.014 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:29.027 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:29.031 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:29.031 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:29.048 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:29.048 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:29.048 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:29.049 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:29.049 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:29.050 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:29.051 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:29.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:29.064 INFO:tasks.rados.rados.0.smithi204.stdout:1597: finishing write tid 1 to smithi20438004-3 2024-09-13T11:33:29.064 INFO:tasks.rados.rados.0.smithi204.stdout:1597: finishing write tid 2 to smithi20438004-3 2024-09-13T11:33:29.064 INFO:tasks.rados.rados.0.smithi204.stdout:1597: finishing write tid 3 to smithi20438004-3 2024-09-13T11:33:29.064 INFO:tasks.rados.rados.0.smithi204.stdout:1597: finishing write tid 4 to smithi20438004-3 2024-09-13T11:33:29.064 INFO:tasks.rados.rados.0.smithi204.stdout:1597: finishing write tid 5 to smithi20438004-3 2024-09-13T11:33:29.064 INFO:tasks.rados.rados.0.smithi204.stdout:1597: oid 3 updating version 0 to 52 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1597: oid 3 version 52 is already newer than 51 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1597: oid 3 version 52 is already newer than 50 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1597: oid 3 updating version 52 to 53 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 53 (ObjNum 329 snap 0 seq_num 329) dirty exists 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1597: left oid 3 (ObjNum 329 snap 0 seq_num 329) 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1598: finishing write tid 1 to smithi20438004-24 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1598: finishing write tid 2 to smithi20438004-24 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1579: done (12 left) 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1585: done (11 left) 2024-09-13T11:33:29.065 INFO:tasks.rados.rados.0.smithi204.stdout:1586: done (10 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1587: done (9 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1589: done (8 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1592: done (7 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1593: done (6 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1594: done (5 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1595: done (4 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1596: done (3 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1597: done (2 left) 2024-09-13T11:33:29.066 INFO:tasks.rados.rados.0.smithi204.stdout:1600: write (excl) oid 25 current snap is 0 2024-09-13T11:33:29.067 INFO:tasks.rados.rados.0.smithi204.stdout:1600: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-13T11:33:29.067 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:29.068 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:29.068 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:29.070 INFO:tasks.rados.rados.0.smithi204.stdout:1600: writing smithi20438004-25 from 763117 to 1473715 tid 1 2024-09-13T11:33:29.077 INFO:tasks.rados.rados.0.smithi204.stdout:1600: writing smithi20438004-25 from 2136128 to 2856229 tid 2 2024-09-13T11:33:29.082 INFO:tasks.rados.rados.0.smithi204.stdout:1600: writing smithi20438004-25 from 3390187 to 3563117 tid 3 2024-09-13T11:33:29.082 INFO:tasks.rados.rados.0.smithi204.stdout:1601: read oid 28 snap -1 2024-09-13T11:33:29.082 INFO:tasks.rados.rados.0.smithi204.stdout:1601: expect (ObjNum 288 snap 0 seq_num 288) 2024-09-13T11:33:29.082 INFO:tasks.rados.rados.0.smithi204.stdout:1602: read oid 5 snap -1 2024-09-13T11:33:29.082 INFO:tasks.rados.rados.0.smithi204.stdout:1602: expect (ObjNum 286 snap 0 seq_num 286) 2024-09-13T11:33:29.082 INFO:tasks.rados.rados.0.smithi204.stdout:1603: read oid 22 snap -1 2024-09-13T11:33:29.082 INFO:tasks.rados.rados.0.smithi204.stdout:1603: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-13T11:33:29.083 INFO:tasks.rados.rados.0.smithi204.stdout:1604: read oid 34 snap -1 2024-09-13T11:33:29.083 INFO:tasks.rados.rados.0.smithi204.stdout:1604: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-13T11:33:29.083 INFO:tasks.rados.rados.0.smithi204.stdout:1605: write oid 9 current snap is 0 2024-09-13T11:33:29.083 INFO:tasks.rados.rados.0.smithi204.stdout:1599: got (22) Invalid argument 2024-09-13T11:33:29.083 INFO:tasks.rados.rados.0.smithi204.stdout:1598: finishing write tid 3 to smithi20438004-24 2024-09-13T11:33:29.083 INFO:tasks.rados.rados.0.smithi204.stdout:1605: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-13T11:33:29.083 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:29.084 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:29.091 INFO:tasks.rados.rados.0.smithi204.stdout:1605: writing smithi20438004-9 from 779924 to 1565771 tid 1 2024-09-13T11:33:29.091 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:29.089+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:29.097 INFO:tasks.rados.rados.0.smithi204.stdout:1605: writing smithi20438004-9 from 2078257 to 2658369 tid 2 2024-09-13T11:33:29.101 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:29.101 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:29.102 INFO:tasks.rados.rados.0.smithi204.stdout:1605: writing smithi20438004-9 from 3301257 to 3579924 tid 3 2024-09-13T11:33:29.102 INFO:tasks.rados.rados.0.smithi204.stdout:1599: done (7 left) 2024-09-13T11:33:29.102 INFO:tasks.rados.rados.0.smithi204.stdout:1606: read oid 48 snap -1 2024-09-13T11:33:29.103 INFO:tasks.rados.rados.0.smithi204.stdout:1606: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:29.103 INFO:tasks.rados.rados.0.smithi204.stdout:1607: set_chunk oid 19 target oid 2024-09-13T11:33:29.114 INFO:tasks.rados.rados.0.smithi204.stdout:1607: set_chunk oid 19 offset: 750080 length: 907264 target oid 5fdc656c15e2b5d52092c38c2406a92f56aa891aa5bb9021116c4d57a56dd560 offset: 0 snap -1 2024-09-13T11:33:29.114 INFO:tasks.rados.rados.0.smithi204.stdout:1608: write oid 38 current snap is 0 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1598: finishing write tid 4 to smithi20438004-24 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1598: finishing write tid 5 to smithi20438004-24 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1598: oid 24 updating version 0 to 115 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1598: oid 24 updating version 115 to 118 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1598: oid 24 version 118 is already newer than 117 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1598: oid 24 version 118 is already newer than 116 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 118 (ObjNum 330 snap 0 seq_num 330) dirty exists 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1598: left oid 24 (ObjNum 330 snap 0 seq_num 330) 2024-09-13T11:33:29.115 INFO:tasks.rados.rados.0.smithi204.stdout:1608: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-13T11:33:29.118 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:29.119 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:29.121 INFO:tasks.rados.rados.0.smithi204.stdout:1608: writing smithi20438004-38 from 796731 to 1257827 tid 1 2024-09-13T11:33:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:29.122 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:29.128 INFO:tasks.rados.rados.0.smithi204.stdout:1608: writing smithi20438004-38 from 1736739 to 2460509 tid 2 2024-09-13T11:33:29.134 INFO:tasks.rados.rados.0.smithi204.stdout:1608: writing smithi20438004-38 from 2928680 to 3596731 tid 3 2024-09-13T11:33:29.134 INFO:tasks.rados.rados.0.smithi204.stdout:1598: done (9 left) 2024-09-13T11:33:29.134 INFO:tasks.rados.rados.0.smithi204.stdout:1609: tier_promote oid 10 2024-09-13T11:33:29.134 INFO:tasks.rados.rados.0.smithi204.stdout:1600: finishing write tid 1 to smithi20438004-25 2024-09-13T11:33:29.134 INFO:tasks.rados.rados.0.smithi204.stdout:1600: finishing write tid 2 to smithi20438004-25 2024-09-13T11:33:29.134 INFO:tasks.rados.rados.0.smithi204.stdout:1600: finishing write tid 3 to smithi20438004-25 2024-09-13T11:33:29.135 INFO:tasks.rados.rados.0.smithi204.stdout:1604: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-13T11:33:29.179 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:29.179 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:29.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:29.179 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:29.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.179 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.768793733000003, 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028959070000000001, 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408205+0000", 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409400+0000", 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011945110000000001 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409410+0000", 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0159999999999999e-05 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409416+0000", 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409467+0000", 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1270999999999998e-05 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409489+0000", 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1682999999999999e-05 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411056+0000", 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015677569999999999 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411100+0000", 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4036999999999999e-05 2024-09-13T11:33:29.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.257889566999999, 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027642500000000002, 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919109+0000", 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3200000000000001e-07 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919918+0000", 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080951299999999999 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919931+0000", 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2662e-05 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.919940+0000", 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3780000000000007e-06 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.920022+0000", 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2057000000000004e-05 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.920044+0000", 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1863999999999999e-05 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.921822+0000", 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017779779999999999 2024-09-13T11:33:29.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.921873+0000", 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0565999999999997e-05 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2315 4.0 e58/56 4:001d90c2:::smithi20438004-28:head v 58'53, mlcod=57'51)", 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.057335924, 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031817899999999999, 2024-09-13T11:33:29.188 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119664+0000", 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8330000000000001e-06 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120926+0000", 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012617710000000001 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120940+0000", 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4544e-05 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120952+0000", 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1676e-05 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121050+0000", 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8177999999999994e-05 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121071+0000", 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1174000000000001e-05 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122794+0000", 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00172215 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122844+0000", 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0464000000000003e-05 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4b088bf8:::smithi20438004-24:head, version: 60'85, data_included: [540672~892928,1630208~155648], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4b088bf8:::smithi20438004-24:head@60'85, size: 3058874, copy_subset: [(0, 3058874)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1785856, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:29.192 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.863808341999999, 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003437201, 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:29.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313191+0000", 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.046e-06 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315519+0000", 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023280670000000001 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315551+0000", 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2440000000000001e-05 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315574+0000", 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2286e-05 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315668+0000", 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929000000000001e-05 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315679+0000", 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1e-05 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.316627+0000", 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094843300000000005 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2354 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'52, mlcod=57'50)", 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:29.196 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.855812191999998, 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058622930000000002, 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321187+0000", 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2200000000000003e-07 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323519+0000", 2024-09-13T11:33:29.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002331891 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323532+0000", 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3494e-05 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323542+0000", 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323612+0000", 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9809999999999997e-05 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323638+0000", 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6526999999999999e-05 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.326990+0000", 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033519270000000002 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327048+0000", 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8158999999999997e-05 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2355 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'53, mlcod=57'50)", 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.852763817, 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028910490000000001, 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.201 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324235+0000", 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6699999999999999e-07 2024-09-13T11:33:29.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324282+0000", 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7437e-05 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324290+0000", 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3170000000000002e-06 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324297+0000", 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.691e-06 2024-09-13T11:33:29.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324944+0000", 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064681799999999996 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324977+0000", 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2591e-05 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327083+0000", 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021065540000000001 2024-09-13T11:33:29.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327125+0000", 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2274000000000002e-05 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2356 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'54, mlcod=57'50)", 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.852690152000001, 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030409059999999999, 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0100000000000001e-07 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324312+0000", 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324318+0000", 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6119999999999998e-06 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324323+0000", 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325363+0000", 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001040116 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325383+0000", 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9644999999999999e-05 2024-09-13T11:33:29.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327310+0000", 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001927205 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327349+0000", 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8930000000000002e-05 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.209 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2513 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'86, mlcod=0'0)", 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.111168163999999, 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034473500000000001, 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8099999999999999e-07 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066800+0000", 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096952400000000004 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066816+0000", 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6010000000000001e-05 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066826+0000", 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0679000000000001e-05 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066970+0000", 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014400900000000001 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.067020+0000", 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9595000000000002e-05 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069220+0000", 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022001640000000001 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069277+0000", 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7188000000000002e-05 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.213 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2515 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'88, mlcod=63'86)", 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.097953557, 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007260702, 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:29.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4300000000000002e-07 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080302+0000", 2024-09-13T11:33:29.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001256889 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080314+0000", 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.134e-05 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080319+0000", 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6699999999999999e-06 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080371+0000", 2024-09-13T11:33:29.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2039000000000001e-05 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080397+0000", 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5996e-05 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086251+0000", 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058537789999999999 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086305+0000", 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4146e-05 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2516 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'89, mlcod=63'86)", 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.096577376999999, 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062074870000000002, 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:29.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7199999999999999e-07 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080426+0000", 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080434+0000", 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2930000000000001e-06 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080443+0000", 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3699999999999995e-06 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081319+0000", 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087608100000000002 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081346+0000", 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7282999999999999e-05 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086595+0000", 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052492759999999998 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086628+0000", 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2836999999999997e-05 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2718 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'58, mlcod=58'56)", 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.053178624000001, 2024-09-13T11:33:29.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039829779999999999, 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:29.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:29.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125730+0000", 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001910213 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125744+0000", 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3326000000000001e-05 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125752+0000", 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125811+0000", 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9582e-05 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125836+0000", 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4482e-05 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127747+0000", 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019112599999999999 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127803+0000", 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5633000000000001e-05 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2719 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'59, mlcod=60'57)", 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:29.229 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.045401364, 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032139159999999998, 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131598+0000", 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7999999999999993e-07 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132663+0000", 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010650010000000001 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132690+0000", 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6970000000000001e-05 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132703+0000", 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.36e-05 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132837+0000", 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013366699999999999 2024-09-13T11:33:29.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132900+0000", 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3331999999999996e-05 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134769+0000", 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018688520000000001 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134811+0000", 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1514000000000003e-05 2024-09-13T11:33:29.233 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3042 4.12 e71/61 4:4b088bf8:::smithi20438004-24:head v 71'99, mlcod=70'97)", 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.060734718000001, 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036713660000000001, 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116264+0000", 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8300000000000001e-07 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117477+0000", 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012129280000000001 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117486+0000", 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7100000000000002e-06 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117496+0000", 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6940000000000007e-06 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117579+0000", 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3207999999999993e-05 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117606+0000", 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6622999999999999e-05 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119900+0000", 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022940899999999999 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119935+0000", 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4929999999999999e-05 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3122 4.12 e73/61 4:4b088bf8:::smithi20438004-24:head v 73'102, mlcod=71'100)", 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.801051576999999, 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031004679999999999, 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.74e-07 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377229+0000", 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001282245 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377237+0000", 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6939999999999995e-06 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377242+0000", 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377289+0000", 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6981999999999998e-05 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377315+0000", 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6604000000000001e-05 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.379013+0000", 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016974959999999999 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.379047+0000", 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4156000000000003e-05 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3247 4.0 e74/56 4:04aeeab9:::smithi20438004-5:head v 74'72, mlcod=74'70)", 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.396418+0000", 2024-09-13T11:33:29.242 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.7805805620000008, 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028934479999999999, 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.396418+0000", 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.396418+0000", 2024-09-13T11:33:29.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.396419+0000", 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5239999999999999e-06 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397566+0000", 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011464260000000001 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397584+0000", 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8138000000000002e-05 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397593+0000", 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4939999999999992e-06 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397655+0000", 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1298999999999997e-05 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397674+0000", 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8998e-05 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.399268+0000", 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015941289999999999 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.399311+0000", 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3439999999999997e-05 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3248 4.0 e74/56 4:04aeeab9:::smithi20438004-5:head v 74'73, mlcod=74'71)", 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.7731941100000004, 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031644770000000002, 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403806+0000", 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6500000000000001e-06 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405022+0000", 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012166569999999999 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405039+0000", 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6942000000000001e-05 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405050+0000", 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0227000000000001e-05 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405112+0000", 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2916999999999994e-05 2024-09-13T11:33:29.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405139+0000", 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6594e-05 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.406922+0000", 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017830540000000001 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.406969+0000", 2024-09-13T11:33:29.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6436000000000001e-05 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3286 4.0 e74/56 4:001d90c2:::smithi20438004-28:head v 74'75, mlcod=74'73)", 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.5819097259999992, 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048258800000000003, 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595090+0000", 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5319999999999999e-06 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596361+0000", 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001270953 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596378+0000", 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7363999999999998e-05 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596386+0000", 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9640000000000003e-06 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596440+0000", 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3544000000000001e-05 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596459+0000", 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8864000000000001e-05 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.599857+0000", 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033986609999999999 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.599914+0000", 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3492 4.12 e76/61 4:4b088bf8:::smithi20438004-24:head v 76'108, mlcod=73'106)", 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.4301070459999998, 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028628780000000001, 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:29.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.85e-07 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748263+0000", 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001371833 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748273+0000", 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0033e-05 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748280+0000", 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3829999999999997e-06 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748349+0000", 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9832000000000005e-05 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748368+0000", 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8368000000000001e-05 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.749720+0000", 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001352341 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.749754+0000", 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3902999999999997e-05 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3493 4.12 e76/61 4:4b088bf8:::smithi20438004-24:head v 76'109, mlcod=73'107)", 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.755369+0000", 2024-09-13T11:33:29.259 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.421628696, 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027875869999999998, 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.755369+0000", 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.755369+0000", 2024-09-13T11:33:29.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.755371+0000", 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.826e-06 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.756600+0000", 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001228488 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.756611+0000", 2024-09-13T11:33:29.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1250000000000001e-05 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.756617+0000", 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.756655+0000", 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8859999999999997e-05 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.756669+0000", 2024-09-13T11:33:29.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3896999999999999e-05 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758128+0000", 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001458988 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758157+0000", 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8711999999999999e-05 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3642 4.12 e76/61 4:4d4c8512:::smithi20438004-29:head v 76'111, mlcod=76'109)", 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.9313857849999998, 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003165579, 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245615+0000", 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246791+0000", 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001175773 2024-09-13T11:33:29.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246805+0000", 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.401e-05 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246813+0000", 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7100000000000007e-06 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246879+0000", 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5889999999999994e-05 2024-09-13T11:33:29.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246909+0000", 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9937000000000002e-05 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.248741+0000", 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018326709999999999 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.248778+0000", 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6738e-05 2024-09-13T11:33:29.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.268 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:29.268 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:29.268 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:29.269 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:29.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:29.270 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:29.268+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:29.283 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:29.342 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:33:29.356 INFO:tasks.rados.rados.0.smithi204.stdout:1600: finishing write tid 4 to smithi20438004-25 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:1600: finishing write tid 5 to smithi20438004-25 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:1600: oid 25 updating version 0 to 107 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:1600: oid 25 version 107 is already newer than 105 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:1600: oid 25 version 107 is already newer than 106 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:1600: oid 25 version 107 is already newer than 104 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 107 (ObjNum 331 snap 0 seq_num 331) dirty exists 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:1600: left oid 25 (ObjNum 331 snap 0 seq_num 331) 2024-09-13T11:33:29.357 INFO:tasks.rados.rados.0.smithi204.stdout:1603: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-13T11:33:29.370 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:29.369+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:29.398 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:29.398 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:29.398 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:29.398 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:29.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:29.458 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:29.472 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:29.471+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:29.518 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:29.519 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:29.522 INFO:tasks.rados.rados.0.smithi204.stdout:1601: expect (ObjNum 288 snap 0 seq_num 288) 2024-09-13T11:33:29.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:29.573 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:29.572+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:29.582 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 668704 2024-09-13T11:33:29.630 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:29.632 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.538744440000002, 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011006991000000001, 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091279+0000", 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.091e-06 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098180+0000", 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069015359999999998 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098194+0000", 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.323e-05 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098202+0000", 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2879999999999993e-06 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098267+0000", 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4733999999999994e-05 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098296+0000", 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102225+0000", 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039281940000000003 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102285+0000", 2024-09-13T11:33:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.481938950999997, 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010863593, 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148085+0000", 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2810000000000001e-06 2024-09-13T11:33:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153254+0000", 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051698430000000004 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153266+0000", 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1973e-05 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153274+0000", 2024-09-13T11:33:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153331+0000", 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6492000000000001e-05 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153396+0000", 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5083e-05 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158906+0000", 2024-09-13T11:33:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055099570000000002 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158947+0000", 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.476726405000001, 2024-09-13T11:33:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096623879999999992, 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157238+0000", 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039414139999999999 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157253+0000", 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5566e-05 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157260+0000", 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.477e-06 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157315+0000", 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4857e-05 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157341+0000", 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162899+0000", 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055583480000000003 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162958+0000", 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9169000000000002e-05 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.467068320000003, 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011410943999999999, 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162955+0000", 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0669999999999999e-06 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167312+0000", 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004357072 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167333+0000", 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0842000000000001e-05 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167341+0000", 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4430000000000003e-06 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167406+0000", 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4245000000000006e-05 2024-09-13T11:33:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167433+0000", 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7500000000000001e-05 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174280+0000", 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068473969999999999 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174365+0000", 2024-09-13T11:33:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4377999999999997e-05 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.454240556999999, 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099135219999999993, 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175783+0000", 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179929+0000", 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041461989999999997 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179943+0000", 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4731e-05 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179953+0000", 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180049+0000", 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6909000000000003e-05 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180073+0000", 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4077e-05 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185641+0000", 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055674330000000001 2024-09-13T11:33:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185695+0000", 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4273e-05 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:258 4.3 4:c717f8db:::smithi20438004-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.416473291000003, 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017162947000000001, 2024-09-13T11:33:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 258 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213551+0000", 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0760000000000001e-06 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213566+0000", 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5441999999999999e-05 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213585+0000", 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8623000000000001e-05 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213600+0000", 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4783999999999999e-05 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213667+0000", 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213747+0000", 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9794000000000002e-05 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228488+0000", 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014740964 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229259+0000", 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077144599999999996 2024-09-13T11:33:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229743+0000", 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048437000000000003 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230652+0000", 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090904600000000005 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230688+0000", 2024-09-13T11:33:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5474e-05 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230712+0000", 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3954000000000001e-05 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:259 4.1 4:8533c921:::smithi20438004-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.412197689000003, 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014548484, 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 259 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217826+0000", 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6360000000000001e-06 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217833+0000", 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8700000000000003e-06 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217844+0000", 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217851+0000", 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217896+0000", 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4808000000000002e-05 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217954+0000", 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7856000000000003e-05 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229813+0000", 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011859533 2024-09-13T11:33:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231353+0000", 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001539503 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232188+0000", 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083500600000000001 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232284+0000", 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6645999999999996e-05 2024-09-13T11:33:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232325+0000", 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0071999999999999e-05 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232373+0000", 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8498999999999998e-05 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:266 4.4 4:2451796c:::smithi20438004-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.407697892999998, 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099464859999999992, 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 266 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222325+0000", 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222330+0000", 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999997e-06 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222341+0000", 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0528e-05 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222347+0000", 2024-09-13T11:33:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6409999999999996e-06 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222406+0000", 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8449000000000002e-05 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222487+0000", 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1167000000000004e-05 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230197+0000", 2024-09-13T11:33:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077105120000000001 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230937+0000", 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000740159 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231296+0000", 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035862399999999999 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232186+0000", 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089002899999999997 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232234+0000", 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7414999999999999e-05 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232271+0000", 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7323000000000002e-05 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:283 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.396569757999998, 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011861159, 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 283 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233453+0000", 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233457+0000", 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1150000000000004e-06 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233465+0000", 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233471+0000", 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3450000000000002e-06 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233526+0000", 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4790000000000002e-05 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233602+0000", 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6253000000000003e-05 2024-09-13T11:33:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237634+0000", 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040318580000000001 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238163+0000", 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052849899999999998 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242711+0000", 2024-09-13T11:33:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045478530000000001 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245235+0000", 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025239160000000002 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245286+0000", 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1739e-05 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245314+0000", 2024-09-13T11:33:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7280999999999999e-05 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:284 4.1 4:8b052c81:::smithi20438004-35:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.396219746, 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011653294, 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 284 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233803+0000", 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4600000000000003e-07 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233808+0000", 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233817+0000", 2024-09-13T11:33:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9870000000000005e-06 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233824+0000", 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8290000000000002e-06 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233874+0000", 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9877999999999999e-05 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233943+0000", 2024-09-13T11:33:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238024+0000", 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040810350000000002 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238440+0000", 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041507799999999999 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242785+0000", 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043457890000000001 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245363+0000", 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025773359999999999 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245416+0000", 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3156999999999999e-05 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245456+0000", 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0000000000000003e-05 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:286 4.5 4:a9b0c9bd:::smithi20438004-37:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.395237985000001, 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013081485, 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 286 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234785+0000", 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234790+0000", 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.651e-06 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234800+0000", 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234807+0000", 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.861e-06 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234868+0000", 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0582999999999999e-05 2024-09-13T11:33:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234967+0000", 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054743730000000003 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241205+0000", 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076331700000000001 2024-09-13T11:33:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241351+0000", 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619399999999999 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247766+0000", 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064151290000000003 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247825+0000", 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8739999999999997e-05 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247866+0000", 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747000000000001e-05 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:290 4.1 4:8d6fdfdb:::smithi20438004-41:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.390683836000001, 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014414148999999999, 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 290 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239339+0000", 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239344+0000", 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2290000000000002e-06 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239354+0000", 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239361+0000", 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.092e-06 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239406+0000", 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126000000000001e-05 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239452+0000", 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6192000000000001e-05 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248478+0000", 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090262209999999992 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249309+0000", 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083057300000000003 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253400+0000", 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040908760000000002 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253662+0000", 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026229500000000001 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253715+0000", 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2836e-05 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253753+0000", 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7526e-05 2024-09-13T11:33:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:292 4.1 4:84fa1f64:::smithi20438004-43:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.390260623000003, 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010951834000000001, 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 292 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8800000000000002e-07 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239765+0000", 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239773+0000", 2024-09-13T11:33:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8229999999999995e-06 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239779+0000", 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9900000000000002e-06 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239831+0000", 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2326e-05 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239910+0000", 2024-09-13T11:33:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8623999999999998e-05 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245574+0000", 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005663927 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.246099+0000", 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052478300000000002 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247963+0000", 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001864166 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250625+0000", 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002662228 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250676+0000", 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0989000000000003e-05 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250713+0000", 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7360000000000001e-05 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:317 4.1 4:8c30e60e:::smithi20438004-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.379136717999998, 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013134860999999999, 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 317 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250892+0000", 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250900+0000", 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250908+0000", 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.337e-06 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250967+0000", 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8201000000000001e-05 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251045+0000", 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299000000000004e-05 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258769+0000", 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007724323 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259599+0000", 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082958200000000004 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260992+0000", 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00139291 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263918+0000", 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029263190000000001 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263971+0000", 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2670999999999998e-05 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264020+0000", 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:330 4.3 4:c7f26bfe:::smithi20438004-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.377913903, 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011943621, 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 330 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252109+0000", 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252116+0000", 2024-09-13T11:33:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9199999999999998e-06 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252129+0000", 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2768e-05 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252138+0000", 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5939999999999994e-06 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252205+0000", 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252280+0000", 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4930000000000003e-05 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259484+0000", 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00720434 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259994+0000", 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051054200000000001 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260333+0000", 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033855000000000002 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263947+0000", 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00361452 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264015+0000", 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7155000000000003e-05 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264052+0000", 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305000000000003e-05 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:333 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.377623706999998, 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011814712, 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 333 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252406+0000", 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252416+0000", 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-05 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252423+0000", 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:33:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252486+0000", 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2805999999999995e-05 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252540+0000", 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3897000000000003e-05 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259138+0000", 2024-09-13T11:33:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065977429999999997 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259785+0000", 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064727700000000003 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264056+0000", 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042706710000000002 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264152+0000", 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5877000000000004e-05 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264180+0000", 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8674999999999999e-05 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264213+0000", 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3015000000000002e-05 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:433 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.353072859999997, 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098551839999999995, 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276950+0000", 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999995e-07 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276955+0000", 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1059999999999999e-06 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276965+0000", 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276977+0000", 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1518000000000001e-05 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277035+0000", 2024-09-13T11:33:29.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8133000000000001e-05 2024-09-13T11:33:29.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277125+0000", 2024-09-13T11:33:29.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0148999999999994e-05 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282235+0000", 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005109461 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:29.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282725+0000", 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049045700000000002 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286634+0000", 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039087929999999998 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286707+0000", 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2692000000000001e-05 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286772+0000", 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5828999999999996e-05 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286805+0000", 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2134000000000001e-05 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:791 4.5 4:a5f8f9e2:::smithi20438004-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:29.730 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.930189949999999, 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019917450999999999, 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 791 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699834+0000", 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699840+0000", 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5989999999999998e-06 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699852+0000", 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699861+0000", 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5259999999999993e-06 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699949+0000", 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8083000000000004e-05 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699953+0000", 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3869999999999997e-06 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701189+0000", 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012354740000000001 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701250+0000", 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1733e-05 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.719750+0000", 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018499504 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.735 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1374 4.1 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.550122821999999, 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011338608999999999, 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1374 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079900+0000", 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-07 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079901+0000", 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079905+0000", 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2039999999999999e-06 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079908+0000", 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8200000000000001e-06 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081697+0000", 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017893080000000001 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081802+0000", 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105231 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.091238+0000", 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094355819999999997 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.739 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1454 4.1 4:84fa1f64:::smithi20438004-43:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 52.794075978999999, 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025598754000000001, 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1454 2024-09-13T11:33:29.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835947+0000", 2024-09-13T11:33:29.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8500000000000002e-07 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835948+0000", 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835952+0000", 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4259999999999999e-06 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:29.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835954+0000", 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8550000000000002e-06 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836019+0000", 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4416000000000001e-05 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836271+0000", 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025171299999999998 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:29.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.861545+0000", 2024-09-13T11:33:29.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025274496 2024-09-13T11:33:29.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.744 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:29.744 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:29.744 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:29.745 INFO:tasks.rados.rados.0.smithi204.stdout:1605: finishing write tid 1 to smithi20438004-9 2024-09-13T11:33:29.745 INFO:tasks.rados.rados.0.smithi204.stdout:1602: expect (ObjNum 286 snap 0 seq_num 286) 2024-09-13T11:33:29.745 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1194184 2024-09-13T11:33:29.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:29.747 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:29.744+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:29.776 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1723898 2024-09-13T11:33:29.794 INFO:tasks.rados.rados.0.smithi204.stdout:1605: finishing write tid 2 to smithi20438004-9 2024-09-13T11:33:29.794 INFO:tasks.rados.rados.0.smithi204.stdout:1606: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:29.826 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2124361 2024-09-13T11:33:29.846 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:29.845+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:29.849 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:29.849 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:29.849 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:29.849 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:29.851 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2044392 2024-09-13T11:33:29.857 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:29.873 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1056227 2024-09-13T11:33:29.905 INFO:tasks.rados.rados.0.smithi204.stdout:1605: finishing write tid 3 to smithi20438004-9 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:1605: finishing write tid 4 to smithi20438004-9 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:1605: finishing write tid 5 to smithi20438004-9 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:1605: oid 9 updating version 0 to 50 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:1605: oid 9 version 50 is already newer than 48 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:1605: oid 9 version 50 is already newer than 49 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:1605: oid 9 updating version 50 to 51 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 51 (ObjNum 332 snap 0 seq_num 332) dirty exists 2024-09-13T11:33:29.906 INFO:tasks.rados.rados.0.smithi204.stdout:1605: left oid 9 (ObjNum 332 snap 0 seq_num 332) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1600: done (9 left) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1601: done (8 left) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1602: done (7 left) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1603: done (6 left) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1604: done (5 left) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1605: done (4 left) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1606: done (3 left) 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1610: delete oid 5 current snap is 0 2024-09-13T11:33:29.907 INFO:tasks.rados.rados.0.smithi204.stdout:1607: finishing set_chunk to oid 19 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 93 (ObjNum 263 snap 0 seq_num 263) dirty exists 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: oid 38 updating version 0 to 133 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: oid 38 version 133 is already newer than 131 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: oid 38 updating version 133 to 134 2024-09-13T11:33:29.908 INFO:tasks.rados.rados.0.smithi204.stdout:1608: oid 38 version 134 is already newer than 132 2024-09-13T11:33:29.909 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 134 (ObjNum 333 snap 0 seq_num 333) dirty exists 2024-09-13T11:33:29.909 INFO:tasks.rados.rados.0.smithi204.stdout:1608: left oid 38 (ObjNum 333 snap 0 seq_num 333) 2024-09-13T11:33:29.909 INFO:tasks.rados.rados.0.smithi204.stdout:1609: got (0) Success 2024-09-13T11:33:29.909 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 114 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-13T11:33:29.909 INFO:tasks.rados.rados.0.smithi204.stdout:1607: done (3 left) 2024-09-13T11:33:29.910 INFO:tasks.rados.rados.0.smithi204.stdout:1608: done (2 left) 2024-09-13T11:33:29.910 INFO:tasks.rados.rados.0.smithi204.stdout:1609: done (1 left) 2024-09-13T11:33:29.910 INFO:tasks.rados.rados.0.smithi204.stdout:1610: done (0 left) 2024-09-13T11:33:29.910 INFO:tasks.rados.rados.0.smithi204.stdout:1611: set_chunk oid 49 target oid 2024-09-13T11:33:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:1611: set_chunk oid 49 offset: 658432 length: 1331712 target oid a5692bc1c86b9782cd7222f07382741ad4e9722d6bf5d7b360ed6a677429e5d1 offset: 0 snap -1 2024-09-13T11:33:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:1612: tier_evict oid 5 2024-09-13T11:33:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:1612: tier_evict oid 5 snap -1 2024-09-13T11:33:29.918 INFO:tasks.rados.rados.0.smithi204.stdout:1613: set_chunk oid 20 target oid 2024-09-13T11:33:29.921 INFO:tasks.rados.rados.0.smithi204.stdout:1613: set_chunk oid 20 offset: 1543168 length: 9728 target oid e319082c23faa07865238e06844382d181ad3cdab44bcfaea7836fd601dba857 offset: 0 snap -1 2024-09-13T11:33:29.921 INFO:tasks.rados.rados.0.smithi204.stdout:1614: read oid 6 snap -1 2024-09-13T11:33:29.921 INFO:tasks.rados.rados.0.smithi204.stdout:1614: expect deleted 2024-09-13T11:33:29.921 INFO:tasks.rados.rados.0.smithi204.stdout:1615: set_chunk oid 41 target oid 2024-09-13T11:33:29.932 INFO:tasks.rados.rados.0.smithi204.stdout:1615: set_chunk oid 41 offset: 709632 length: 1531904 target oid 3c8c29a5fd26bb117667ecf79e23b75c3b531b56f7971d094df3f7365da1d6b4 offset: 0 snap -1 2024-09-13T11:33:29.933 INFO:tasks.rados.rados.0.smithi204.stdout:1616: delete oid 42 current snap is 0 2024-09-13T11:33:29.933 INFO:tasks.rados.rados.0.smithi204.stdout:1612: got (22) Invalid argument 2024-09-13T11:33:29.933 INFO:tasks.rados.rados.0.smithi204.stdout:1611: finishing set_chunk to oid 49 2024-09-13T11:33:29.933 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 37 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-13T11:33:29.933 INFO:tasks.rados.rados.0.smithi204.stdout:1613: finishing set_chunk to oid 20 2024-09-13T11:33:29.933 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 1543168 length: 9728 2024-09-13T11:33:29.933 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 0 (ObjNum 183 snap 0 seq_num 183) dirty exists 2024-09-13T11:33:29.934 INFO:tasks.rados.rados.0.smithi204.stdout:1611: done (5 left) 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1612: done (4 left) 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1613: done (3 left) 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1614: done (2 left) 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1616: done (1 left) 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1617: tier_evict oid 50 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1617: tier_evict oid 50 snap -1 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1618: read oid 36 snap -1 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1618: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1619: set_chunk oid 37 target oid 2024-09-13T11:33:29.935 INFO:tasks.rados.rados.0.smithi204.stdout:1619: set_chunk oid 37 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:29.936 INFO:tasks.rados.rados.0.smithi204.stdout:1620: write (excl) oid 39 current snap is 0 2024-09-13T11:33:29.936 INFO:tasks.rados.rados.0.smithi204.stdout:1620: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-13T11:33:29.938 INFO:tasks.rados.rados.0.smithi204.stdout:1620: writing smithi20438004-39 from 447152 to 850348 tid 1 2024-09-13T11:33:29.945 INFO:tasks.rados.rados.0.smithi204.stdout:1620: writing smithi20438004-39 from 1512185 to 2099635 tid 2 2024-09-13T11:33:29.946 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:29.945+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:29.953 INFO:tasks.rados.rados.0.smithi204.stdout:1620: writing smithi20438004-39 from 2727302 to 3408463 tid 3 2024-09-13T11:33:29.953 INFO:tasks.rados.rados.0.smithi204.stdout:1621: read oid 32 snap -1 2024-09-13T11:33:29.953 INFO:tasks.rados.rados.0.smithi204.stdout:1621: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:29.953 INFO:tasks.rados.rados.0.smithi204.stdout:1622: set_chunk oid 13 target oid 2024-09-13T11:33:29.961 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:29.961 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:29.961 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:29.961 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:29.961 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:29.963 INFO:tasks.rados.rados.0.smithi204.stdout:1622: set_chunk oid 13 offset: 784384 length: 1361920 target oid 93d9f5eac63111d714d2ad72f7d8cb9808a1dd4ede02886e4a268512b4688dd9 offset: 0 snap -1 2024-09-13T11:33:29.963 INFO:tasks.rados.rados.0.smithi204.stdout:1623: set_chunk oid 40 target oid 2024-09-13T11:33:29.970 INFO:tasks.rados.rados.0.smithi204.stdout:1623: set_chunk oid 40 offset: 1393152 length: 814592 target oid 2ae759fc85d8bd0969d4acd040d9b0925b4b07f8e2f3352e6f9e8bee13e1d5b2 offset: 0 snap -1 2024-09-13T11:33:29.971 INFO:tasks.rados.rados.0.smithi204.stdout:1624: read oid 43 snap -1 2024-09-13T11:33:29.971 INFO:tasks.rados.rados.0.smithi204.stdout:1624: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:29.971 INFO:tasks.rados.rados.0.smithi204.stdout:1625: set_chunk oid 46 target oid 2024-09-13T11:33:29.971 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:29.980 INFO:tasks.rados.rados.0.smithi204.stdout:1625: set_chunk oid 46 offset: 1214464 length: 1342464 target oid 2888ae0d4dde2c1fce9cf5c91c1ecf8ed9de34c478d797cd65edddca34581468 offset: 0 snap -1 2024-09-13T11:33:29.980 INFO:tasks.rados.rados.0.smithi204.stdout:1626: write oid 38 current snap is 0 2024-09-13T11:33:29.980 INFO:tasks.rados.rados.0.smithi204.stdout:1619: finishing set_chunk to oid 37 2024-09-13T11:33:29.980 INFO:tasks.rados.rados.0.smithi204.stdout:1619: got expected ENOENT (src dne) 2024-09-13T11:33:29.980 INFO:tasks.rados.rados.0.smithi204.stdout:1626: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-13T11:33:29.984 INFO:tasks.rados.rados.0.smithi204.stdout:1626: writing smithi20438004-38 from 463959 to 942404 tid 1 2024-09-13T11:33:29.992 INFO:tasks.rados.rados.0.smithi204.stdout:1626: writing smithi20438004-38 from 1570667 to 2301775 tid 2 2024-09-13T11:33:29.999 INFO:tasks.rados.rados.0.smithi204.stdout:1626: writing smithi20438004-38 from 3038372 to 3530805 tid 3 2024-09-13T11:33:29.999 INFO:tasks.rados.rados.0.smithi204.stdout:1619: done (10 left) 2024-09-13T11:33:29.999 INFO:tasks.rados.rados.0.smithi204.stdout:1627: set_chunk oid 30 target oid 2024-09-13T11:33:30.009 INFO:tasks.rados.rados.0.smithi204.stdout:1627: set_chunk oid 30 offset: 574464 length: 1075712 target oid 2b6136a647ef1d0096391558236ea273cf3f7061a3538091ae217a9d92d46d49 offset: 0 snap -1 2024-09-13T11:33:30.009 INFO:tasks.rados.rados.0.smithi204.stdout:1628: set_chunk oid 25 target oid 2024-09-13T11:33:30.012 INFO:tasks.rados.rados.0.smithi204.stdout:1628: set_chunk oid 25 offset: 1684480 length: 128000 target oid eec19bc6af0b3b6dfb97a08782c65f4bb3c3203e789a015d2008b0d689ad08be offset: 0 snap -1 2024-09-13T11:33:30.012 INFO:tasks.rados.rados.0.smithi204.stdout:1629: read oid 21 snap -1 2024-09-13T11:33:30.012 INFO:tasks.rados.rados.0.smithi204.stdout:1629: expect deleted 2024-09-13T11:33:30.012 INFO:tasks.rados.rados.0.smithi204.stdout:1630: read oid 42 snap -1 2024-09-13T11:33:30.012 INFO:tasks.rados.rados.0.smithi204.stdout:1630: expect deleted 2024-09-13T11:33:30.012 INFO:tasks.rados.rados.0.smithi204.stdout:1631: read oid 18 snap -1 2024-09-13T11:33:30.013 INFO:tasks.rados.rados.0.smithi204.stdout:1631: expect deleted 2024-09-13T11:33:30.013 INFO:tasks.rados.rados.0.smithi204.stdout:1615: finishing set_chunk to oid 41 2024-09-13T11:33:30.013 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 119 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-13T11:33:30.013 INFO:tasks.rados.rados.0.smithi204.stdout:1617: got (22) Invalid argument 2024-09-13T11:33:30.013 INFO:tasks.rados.rados.0.smithi204.stdout:1618: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:30.031 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1393156 2024-09-13T11:33:30.047 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:30.046+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 83.731716496000004, 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079674329999999995, 2024-09-13T11:33:30.077 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4344", 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343169+0000", 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343177+0000", 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4819999999999994e-06 2024-09-13T11:33:30.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343189+0000", 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2306999999999999e-05 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343199+0000", 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343279+0000", 2024-09-13T11:33:30.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0025999999999996e-05 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343289+0000", 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0518e-05 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348428+0000", 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051380779999999999 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348510+0000", 2024-09-13T11:33:30.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2226999999999997e-05 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348598+0000", 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8722999999999997e-05 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349014+0000", 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041537900000000001 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349279+0000", 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026517700000000001 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351081+0000", 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001802292 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351110+0000", 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8651000000000001e-05 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351132+0000", 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069000000000002e-05 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:21 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'4, mlcod=38'2)", 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.561733443999998, 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002707297, 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.084 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513149+0000", 2024-09-13T11:33:30.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.322e-06 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514452+0000", 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303153 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514468+0000", 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6045999999999998e-05 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514478+0000", 2024-09-13T11:33:30.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4960000000000002e-06 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514531+0000", 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3442000000000001e-05 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514546+0000", 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5211999999999999e-05 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515821+0000", 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001274854 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515855+0000", 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3772000000000001e-05 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:30.088 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.053506945999999, 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002340637, 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:30.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021376+0000", 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.013e-06 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022454+0000", 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010781110000000001 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022469+0000", 2024-09-13T11:33:30.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4199000000000001e-05 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022477+0000", 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022532+0000", 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5636000000000001e-05 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022548+0000", 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5769000000000001e-05 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023686+0000", 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011379929999999999 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023715+0000", 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8659000000000001e-05 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.092 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:182 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'8, mlcod=38'6)", 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.047441272999997, 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022898689999999999, 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:30.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027441+0000", 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7599999999999996e-07 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028556+0000", 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011154439999999999 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028566+0000", 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8269999999999993e-06 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028571+0000", 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8550000000000001e-06 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028614+0000", 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028629+0000", 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029703+0000", 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001073898 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029730+0000", 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6993999999999999e-05 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:190 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'11, mlcod=38'9)", 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.017613544, 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022963139999999998, 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.097 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:30.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058460+0000", 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192238 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058470+0000", 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058475+0000", 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5399999999999997e-06 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058514+0000", 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8887999999999998e-05 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058526+0000", 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1683e-05 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059537+0000", 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001011907 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059564+0000", 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:30.101 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 62.680430237000003, 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048595679999999999, 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:30.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394452+0000", 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397390+0000", 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002937977 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397401+0000", 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1172000000000001e-05 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397407+0000", 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397452+0000", 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4705e-05 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397467+0000", 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5622000000000001e-05 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399273+0000", 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001805823 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399310+0000", 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7512999999999998e-05 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1363 4.2 e43/31 4:4ba9f189:::smithi20438004-14:head v 43'50, mlcod=43'48)", 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 54.029297804000002, 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021379849999999998, 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045585+0000", 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.604e-06 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045958+0000", 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000372672 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045978+0000", 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0262999999999999e-05 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045991+0000", 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2811e-05 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046079+0000", 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8667999999999999e-05 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046103+0000", 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4131e-05 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047664+0000", 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001560098 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047721+0000", 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7738000000000002e-05 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1410 4.4 e43/41 4:2451796c:::smithi20438004-17:head v 43'44, mlcod=43'42)", 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.946325295999998, 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022888489999999999, 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.110 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128557+0000", 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3939999999999999e-06 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129014+0000", 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045637100000000001 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129032+0000", 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8685000000000002e-05 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129042+0000", 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129100+0000", 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8238e-05 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129119+0000", 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8258e-05 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130809+0000", 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001690045 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130845+0000", 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5991000000000002e-05 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:30.114 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.043735390000002, 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071247150000000002, 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:30.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031147+0000", 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.579e-06 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036431+0000", 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00528394 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036449+0000", 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7329e-05 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036457+0000", 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036529+0000", 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1506000000000003e-05 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036551+0000", 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2154000000000002e-05 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038234+0000", 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001683192 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038270+0000", 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6408000000000001e-05 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1679 4.4 4.9c0290d4 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 48.015677685999997, 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47635528100000002, 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1679 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.119 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059205+0000", 2024-09-13T11:33:30.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.812e-06 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059211+0000", 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1469999999999998e-06 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059222+0000", 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0560000000000001e-05 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059233+0000", 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0591000000000001e-05 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059286+0000", 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519e-05 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059293+0000", 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4690000000000002e-06 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535521+0000", 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47622830100000002 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535559+0000", 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7882000000000001e-05 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:30.123 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.672430056000003, 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002436952, 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402453+0000", 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.632e-06 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403268+0000", 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081553400000000003 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403289+0000", 2024-09-13T11:33:30.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0353e-05 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403298+0000", 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4760000000000004e-06 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403369+0000", 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0858000000000005e-05 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403397+0000", 2024-09-13T11:33:30.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8518e-05 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404841+0000", 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001443228 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404888+0000", 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7352999999999999e-05 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.666903660000003, 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003287178, 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407979+0000", 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409585+0000", 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016060479999999999 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409598+0000", 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3027000000000001e-05 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409606+0000", 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2190000000000004e-06 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409676+0000", 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9791999999999999e-05 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409707+0000", 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1692000000000001e-05 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411223+0000", 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001515705 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411265+0000", 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1649000000000001e-05 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1705 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'62, mlcod=0'0)", 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.659299259999997, 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003138161, 2024-09-13T11:33:30.132 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415583+0000", 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9699999999999995e-07 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416900+0000", 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001317251 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416926+0000", 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5622e-05 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416942+0000", 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6741000000000001e-05 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417014+0000", 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1298999999999996e-05 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417042+0000", 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.851e-05 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418673+0000", 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016310949999999999 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418720+0000", 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6845999999999998e-05 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.136 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1706 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'63, mlcod=45'61)", 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 44.653967063000003, 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003090597, 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420915+0000", 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-06 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422306+0000", 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001390753 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422317+0000", 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422322+0000", 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4360000000000001e-06 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422379+0000", 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7003000000000003e-05 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422405+0000", 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6424e-05 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.423958+0000", 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015526680000000001 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.424005+0000", 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6619000000000002e-05 2024-09-13T11:33:30.140 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1919 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'63, mlcod=53'61)", 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 40.108767084999997, 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002758908, 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966116+0000", 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0370000000000001e-06 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967135+0000", 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010185260000000001 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967149+0000", 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3903999999999999e-05 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967157+0000", 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.405e-06 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967255+0000", 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8441000000000002e-05 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967280+0000", 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5079000000000002e-05 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968828+0000", 2024-09-13T11:33:30.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001547936 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968873+0000", 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4579999999999997e-05 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 40.100816362000003, 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028733999999999999, 2024-09-13T11:33:30.145 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974066+0000", 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975332+0000", 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012659349999999999 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975345+0000", 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3402e-05 2024-09-13T11:33:30.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975351+0000", 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4450000000000001e-06 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975438+0000", 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6687999999999994e-05 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975457+0000", 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8913e-05 2024-09-13T11:33:30.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976901+0000", 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014437180000000001 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976938+0000", 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7441999999999998e-05 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.155682597000002, 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022722770000000001, 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919200+0000", 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9000000000000005e-07 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919903+0000", 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070307200000000001 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919915+0000", 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2482e-05 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919922+0000", 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5459999999999997e-06 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919988+0000", 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6377999999999994e-05 2024-09-13T11:33:30.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.920027+0000", 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8649999999999998e-05 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921419+0000", 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013924530000000001 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921471+0000", 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1706000000000003e-05 2024-09-13T11:33:30.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2077 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'67, mlcod=53'65)", 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.147539965999997, 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026288430000000001, 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927342+0000", 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928463+0000", 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011212640000000001 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928476+0000", 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3147e-05 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928484+0000", 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928549+0000", 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5001999999999999e-05 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928565+0000", 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6124999999999999e-05 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929914+0000", 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001348859 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929970+0000", 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5912000000000003e-05 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:3290 4.4 e74/56 4:2451796c:::smithi20438004-17:head v 74'77, mlcod=73'75)", 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.468535627, 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025948210000000002, 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.158 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606347+0000", 2024-09-13T11:33:30.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607289+0000", 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094240700000000003 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607322+0000", 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3040999999999998e-05 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607350+0000", 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7331999999999999e-05 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607464+0000", 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114826 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607491+0000", 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6271e-05 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.608888+0000", 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013977309999999999 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.608940+0000", 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1826000000000002e-05 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:3291 4.4 e74/56 4:2451796c:::smithi20438004-17:head v 74'78, mlcod=73'76)", 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:30.162 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.463418329, 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022192150000000001, 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611464+0000", 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612224+0000", 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076036799999999998 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612238+0000", 2024-09-13T11:33:30.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.342e-05 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612244+0000", 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4949999999999996e-06 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612295+0000", 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0392000000000001e-05 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612314+0000", 2024-09-13T11:33:30.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9568e-05 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.613631+0000", 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013171019999999999 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.613682+0000", 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0841000000000003e-05 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.166 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.167 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:30.167 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:30.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:30.169 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:30.167+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:30.243 INFO:tasks.rados.rados.0.smithi204.stdout:1620: finishing write tid 1 to smithi20438004-39 2024-09-13T11:33:30.243 INFO:tasks.rados.rados.0.smithi204.stdout:1620: finishing write tid 2 to smithi20438004-39 2024-09-13T11:33:30.243 INFO:tasks.rados.rados.0.smithi204.stdout:1621: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:30.262 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:30.269 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:30.267+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:30.315 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:30.315 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:30.315 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:30.315 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:30.324 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:30.328 INFO:tasks.rados.rados.0.smithi204.stdout:1620: finishing write tid 3 to smithi20438004-39 2024-09-13T11:33:30.328 INFO:tasks.rados.rados.0.smithi204.stdout:1620: finishing write tid 4 to smithi20438004-39 2024-09-13T11:33:30.328 INFO:tasks.rados.rados.0.smithi204.stdout:1620: finishing write tid 5 to smithi20438004-39 2024-09-13T11:33:30.328 INFO:tasks.rados.rados.0.smithi204.stdout:1620: oid 39 updating version 0 to 110 2024-09-13T11:33:30.328 INFO:tasks.rados.rados.0.smithi204.stdout:1620: oid 39 updating version 110 to 112 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:1620: oid 39 version 112 is already newer than 109 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:1620: oid 39 version 112 is already newer than 111 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 112 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:1620: left oid 39 (ObjNum 336 snap 0 seq_num 336) 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:1622: finishing set_chunk to oid 13 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 53 (ObjNum 211 snap 0 seq_num 211) dirty exists 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:1624: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:30.329 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:30.369 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:30.368+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:30.469 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:30.469 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:30.469 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:30.469 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:30.469 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:30.470 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:30.469+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:30.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:30.544 INFO:tasks.rados.rados.0.smithi204.stdout:1623: finishing set_chunk to oid 40 2024-09-13T11:33:30.544 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 40 v 103 (ObjNum 320 snap 0 seq_num 320) dirty exists 2024-09-13T11:33:30.544 INFO:tasks.rados.rados.0.smithi204.stdout:1625: finishing set_chunk to oid 46 2024-09-13T11:33:30.544 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 46 set_chunk returned error code -95 offset: 1214464 length: 1342464 2024-09-13T11:33:30.544 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 0 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-09-13T11:33:30.544 INFO:tasks.rados.rados.0.smithi204.stdout:1626: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:30.544 INFO:tasks.rados.rados.0.smithi204.stdout:1632: read oid 27 snap -1 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1632: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1615: done (15 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1617: done (14 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1618: done (13 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1620: done (12 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1621: done (11 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1622: done (10 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1623: done (9 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1624: done (8 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1625: done (7 left) 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1633: tier_evict oid 16 2024-09-13T11:33:30.545 INFO:tasks.rados.rados.0.smithi204.stdout:1633: tier_evict oid 16 snap -1 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: oid 38 updating version 0 to 138 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: oid 38 version 138 is already newer than 135 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: oid 38 version 138 is already newer than 136 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: oid 38 version 138 is already newer than 137 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 138 (ObjNum 337 snap 0 seq_num 337) dirty exists 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1626: left oid 38 (ObjNum 337 snap 0 seq_num 337) 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:1627: finishing set_chunk to oid 30 2024-09-13T11:33:30.546 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 574464 length: 1075712 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 0 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:1626: done (7 left) 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:1627: done (6 left) 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:1634: read oid 23 snap -1 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:1634: expect deleted 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:1635: write oid 33 current snap is 0 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:1628: finishing set_chunk to oid 25 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 107 (ObjNum 331 snap 0 seq_num 331) dirty exists 2024-09-13T11:33:30.547 INFO:tasks.rados.rados.0.smithi204.stdout:1635: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-13T11:33:30.548 INFO:tasks.rados.rados.0.smithi204.stdout:1635: writing smithi20438004-33 from 480766 to 1034460 tid 1 2024-09-13T11:33:30.555 INFO:tasks.rados.rados.0.smithi204.stdout:1635: writing smithi20438004-33 from 1629149 to 2220268 tid 2 2024-09-13T11:33:30.561 INFO:tasks.rados.rados.0.smithi204.stdout:1635: writing smithi20438004-33 from 2782148 to 3202206 tid 3 2024-09-13T11:33:30.561 INFO:tasks.rados.rados.0.smithi204.stdout:1628: done (7 left) 2024-09-13T11:33:30.561 INFO:tasks.rados.rados.0.smithi204.stdout:1629: done (6 left) 2024-09-13T11:33:30.561 INFO:tasks.rados.rados.0.smithi204.stdout:1630: done (5 left) 2024-09-13T11:33:30.561 INFO:tasks.rados.rados.0.smithi204.stdout:1631: done (4 left) 2024-09-13T11:33:30.561 INFO:tasks.rados.rados.0.smithi204.stdout:1636: set_chunk oid 4 target oid 2024-09-13T11:33:30.571 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:30.570+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:30.573 INFO:tasks.rados.rados.0.smithi204.stdout:1636: set_chunk oid 4 offset: 603136 length: 1722880 target oid 5988c35938008c9925b835687478f20dd58e8a1021bee9715344132ce117567f offset: 0 snap -1 2024-09-13T11:33:30.573 INFO:tasks.rados.rados.0.smithi204.stdout:1637: tier_flush oid 22 2024-09-13T11:33:30.573 INFO:tasks.rados.rados.0.smithi204.stdout:1637: tier_flush oid 22 snap -1 2024-09-13T11:33:30.573 INFO:tasks.rados.rados.0.smithi204.stdout:1633: got (22) Invalid argument 2024-09-13T11:33:30.573 INFO:tasks.rados.rados.0.smithi204.stdout:1633: done (5 left) 2024-09-13T11:33:30.573 INFO:tasks.rados.rados.0.smithi204.stdout:1638: set_chunk oid 41 target oid 2024-09-13T11:33:30.580 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:30.580 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:206 3.1 e38/30 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.486421127, 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012415890000000001, 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092192+0000", 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.63e-07 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100358+0000", 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081664059999999993 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100384+0000", 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5287e-05 2024-09-13T11:33:30.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100401+0000", 2024-09-13T11:33:30.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7744000000000002e-05 2024-09-13T11:33:30.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100478+0000", 2024-09-13T11:33:30.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6769999999999999e-05 2024-09-13T11:33:30.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100508+0000", 2024-09-13T11:33:30.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0134e-05 2024-09-13T11:33:30.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104514+0000", 2024-09-13T11:33:30.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040060590000000002 2024-09-13T11:33:30.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104607+0000", 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3027000000000002e-05 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:201 3.1 e38/30 3:b265d015:::smithi20438004-2 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.478175788000001, 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012387119, 2024-09-13T11:33:30.589 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100438+0000", 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1000000000000001e-06 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108964+0000", 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085255959999999999 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108984+0000", 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0466e-05 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108996+0000", 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2149e-05 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109097+0000", 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010042799999999999 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109121+0000", 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.455e-05 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112773+0000", 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036519780000000002 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112824+0000", 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0852e-05 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.469554316999997, 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096300529999999995, 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.594 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.595 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.595 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:30.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109060+0000", 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116086+0000", 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070260239999999996 2024-09-13T11:33:30.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116100+0000", 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4209999999999999e-05 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116109+0000", 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6569999999999992e-06 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116160+0000", 2024-09-13T11:33:30.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0470000000000003e-05 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116191+0000", 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1494e-05 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118639+0000", 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024481030000000001 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118688+0000", 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8866999999999998e-05 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.600 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.462482522000002, 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010922463, 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116131+0000", 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6899999999999998e-07 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124407+0000", 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082764919999999999 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124422+0000", 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5404999999999999e-05 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124432+0000", 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8910000000000007e-06 2024-09-13T11:33:30.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124485+0000", 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2750999999999997e-05 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124508+0000", 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3076999999999999e-05 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127004+0000", 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024962930000000001 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127053+0000", 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8084999999999998e-05 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.448818424000002, 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091903760000000001, 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129795+0000", 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8700000000000001e-07 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135325+0000", 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055302190000000003 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135340+0000", 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135351+0000", 2024-09-13T11:33:30.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1358999999999999e-05 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135418+0000", 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000002e-05 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135447+0000", 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8415000000000002e-05 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138936+0000", 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034896800000000002 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138985+0000", 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8338999999999999e-05 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.610 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.438623352, 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010981263, 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139990+0000", 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2e-07 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147357+0000", 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073669950000000003 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147376+0000", 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9453000000000002e-05 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147388+0000", 2024-09-13T11:33:30.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1527e-05 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147472+0000", 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4630999999999997e-05 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147493+0000", 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0687e-05 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150899+0000", 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034054739999999999 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150971+0000", 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1976000000000003e-05 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.431172343999997, 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015742025999999999, 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:30.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147441+0000", 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7700000000000004e-07 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157019+0000", 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095782090000000007 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157057+0000", 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157072+0000", 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5082999999999999e-05 2024-09-13T11:33:30.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157157+0000", 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5065e-05 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157188+0000", 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0920000000000002e-05 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163133+0000", 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059446890000000004 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163182+0000", 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9564999999999997e-05 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:30.620 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.421100256000003, 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015556076, 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157514+0000", 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167722+0000", 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010208016 2024-09-13T11:33:30.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167740+0000", 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7668000000000001e-05 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167754+0000", 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4647999999999999e-05 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167831+0000", 2024-09-13T11:33:30.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7229000000000002e-05 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167855+0000", 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3467000000000001e-05 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173001+0000", 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051458160000000001 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173068+0000", 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7720999999999998e-05 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.410831764000001, 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011370243, 2024-09-13T11:33:30.625 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3499999999999996e-07 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175562+0000", 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077811060000000003 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175578+0000", 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175587+0000", 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175668+0000", 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1224999999999996e-05 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175695+0000", 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6505000000000001e-05 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179075+0000", 2024-09-13T11:33:30.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003380508 2024-09-13T11:33:30.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179151+0000", 2024-09-13T11:33:30.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5927000000000007e-05 2024-09-13T11:33:30.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.630 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.403005102999998, 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017514140000000001, 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.631 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:30.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188284+0000", 2024-09-13T11:33:30.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012675702 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188299+0000", 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5122e-05 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188308+0000", 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9819999999999997e-06 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188367+0000", 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8774999999999999e-05 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188387+0000", 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0649000000000002e-05 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193056+0000", 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046684830000000002 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193122+0000", 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5790999999999994e-05 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.390280124999997, 2024-09-13T11:33:30.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015855429000000001, 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:30.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188333+0000", 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-07 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199600+0000", 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011266717000000001 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199613+0000", 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3682e-05 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199620+0000", 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1339999999999998e-06 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199676+0000", 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5689000000000002e-05 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199708+0000", 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2082999999999998e-05 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204123+0000", 2024-09-13T11:33:30.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004414734 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204188+0000", 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:30.641 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.378971587999999, 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092010560000000009, 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199642+0000", 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9599999999999999e-07 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205539+0000", 2024-09-13T11:33:30.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058972290000000004 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205553+0000", 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4577e-05 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205564+0000", 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0254e-05 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205630+0000", 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6057999999999997e-05 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205658+0000", 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7979000000000001e-05 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208792+0000", 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031342499999999999 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208842+0000", 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0213000000000002e-05 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:245 3.1 e38/30 3:adf25eb4:::smithi20438004-46 low_tier:head v 38'15, mlcod=38'11)", 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.373025259000002, 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011146911000000001, 2024-09-13T11:33:30.646 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205588+0000", 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213344+0000", 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077554440000000002 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213359+0000", 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5455999999999999e-05 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213369+0000", 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9890000000000004e-06 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213453+0000", 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4153999999999994e-05 2024-09-13T11:33:30.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213479+0000", 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5831999999999999e-05 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216694+0000", 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032149990000000001 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216734+0000", 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0102999999999999e-05 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:260 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.359254237999998, 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011239344, 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.651 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 260 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219360+0000", 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219366+0000", 2024-09-13T11:33:30.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7179999999999999e-06 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219376+0000", 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9990000000000003e-06 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219382+0000", 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8150000000000002e-06 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219428+0000", 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5751999999999998e-05 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4119000000000001e-05 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227887+0000", 2024-09-13T11:33:30.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083953989999999996 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228998+0000", 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001110764 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229290+0000", 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029145400000000002 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230489+0000", 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001199786 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230548+0000", 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8900999999999997e-05 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230598+0000", 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9391999999999997e-05 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.657 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:249 3.1 e38/30 3:81faf980:::smithi20438004-50 low_tier:head v 38'21, mlcod=38'18)", 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.355978102000002, 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091134460000000007, 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.68e-07 2024-09-13T11:33:30.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227843+0000", 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005207494 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227853+0000", 2024-09-13T11:33:30.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117e-05 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227860+0000", 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4880000000000001e-06 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227912+0000", 2024-09-13T11:33:30.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1900000000000001e-05 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227933+0000", 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0474000000000001e-05 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231709+0000", 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037762759999999999 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231748+0000", 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9029000000000002e-05 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:278 4.2 4:4d4c8512:::smithi20438004-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.347618502000003, 2024-09-13T11:33:30.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091148359999999994, 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230995+0000", 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4200000000000004e-07 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231000+0000", 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3290000000000001e-06 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231017+0000", 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6242000000000001e-05 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231025+0000", 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231094+0000", 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9554999999999994e-05 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231171+0000", 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7082999999999994e-05 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236799+0000", 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056282260000000001 2024-09-13T11:33:30.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237539+0000", 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073930700000000003 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237574+0000", 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5191999999999998e-05 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240013+0000", 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002439185 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240070+0000", 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6650999999999997e-05 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240109+0000", 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9131999999999997e-05 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:287 4.2 4:4297cb20:::smithi20438004-38:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.343683030000001, 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013317569, 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.670 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 287 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234931+0000", 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.192e-06 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234936+0000", 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0640000000000001e-06 2024-09-13T11:33:30.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234953+0000", 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7292e-05 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234961+0000", 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:33:30.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235026+0000", 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5073000000000005e-05 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235111+0000", 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4635999999999994e-05 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:30.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242149+0000", 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070384990000000001 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069074700000000002 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242936+0000", 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5730000000000007e-05 2024-09-13T11:33:30.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248138+0000", 2024-09-13T11:33:30.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052018330000000003 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248197+0000", 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9512000000000002e-05 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248247+0000", 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.677 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:310 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.329756662000001, 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095515259999999994, 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:30.678 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 310 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248857+0000", 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-06 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248863+0000", 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6529999999999999e-06 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248874+0000", 2024-09-13T11:33:30.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.093e-05 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248887+0000", 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3009e-05 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248932+0000", 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5445999999999998e-05 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249025+0000", 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3098999999999998e-05 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251503+0000", 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024772629999999999 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252148+0000", 2024-09-13T11:33:30.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064507900000000001 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252433+0000", 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028559100000000002 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258304+0000", 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058703100000000001 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258361+0000", 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7658999999999998e-05 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258407+0000", 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6319999999999997e-05 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.684 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:313 4.2 4:4ba9f189:::smithi20438004-14:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.328877955000003, 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013958247999999999, 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 313 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249736+0000", 2024-09-13T11:33:30.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9999999999999996e-07 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249741+0000", 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5929999999999996e-06 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249750+0000", 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7369999999999995e-06 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249757+0000", 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5360000000000004e-06 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249806+0000", 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9018999999999999e-05 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249882+0000", 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5943000000000001e-05 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258629+0000", 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087464589999999998 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259299+0000", 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067041799999999999 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259523+0000", 2024-09-13T11:33:30.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223964 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263636+0000", 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041130539999999997 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263669+0000", 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2849999999999999e-05 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263693+0000", 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3774999999999999e-05 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1437 4.2 4:4b088bf8:::smithi20438004-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 53.774637568999999, 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014642921, 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:30.691 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1437 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803976+0000", 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.35e-06 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803984+0000", 2024-09-13T11:33:30.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2080000000000003e-06 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803995+0000", 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804001+0000", 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2870000000000001e-06 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804096+0000", 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5424e-05 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804165+0000", 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8596000000000005e-05 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.818618+0000", 2024-09-13T11:33:30.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452906999999999 2024-09-13T11:33:30.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.696 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:30.696 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:30.696 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:30.698 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:30.696+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:30.699 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:30.700 INFO:tasks.rados.rados.0.smithi204.stdout:1638: set_chunk oid 41 offset: 489984 length: 1103360 target oid 72165e0b25de4a0b99fe20ce950b041a6c5ee572d5dfe3df9a6243f2e5ddbaf1 offset: 0 snap -1 2024-09-13T11:33:30.700 INFO:tasks.rados.rados.0.smithi204.stdout:1639: read oid 8 snap -1 2024-09-13T11:33:30.700 INFO:tasks.rados.rados.0.smithi204.stdout:1639: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1640: write (excl) oid 14 current snap is 0 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1640: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1640: writing smithi20438004-14 from 497573 to 1126516 tid 1 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1640: writing smithi20438004-14 from 1687631 to 2422408 tid 2 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1640: writing smithi20438004-14 from 3093218 to 3697573 tid 3 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1641: set_chunk oid 49 target oid 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1641: set_chunk oid 49 offset: 1803776 length: 471552 target oid 2e9bc2fc883052427e63e9fd2e29f6a4cb32560de4adde5c87c6bf22c19b4b0c offset: 0 snap -1 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1642: write (excl) oid 34 current snap is 0 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1642: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1642: writing smithi20438004-34 from 514380 to 1218572 tid 1 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1642: writing smithi20438004-34 from 1629760 to 2108195 tid 2 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1642: writing smithi20438004-34 from 2604288 to 3046890 tid 3 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1643: tier_promote oid 2 2024-09-13T11:33:30.701 INFO:tasks.rados.rados.0.smithi204.stdout:1634: done (10 left) 2024-09-13T11:33:30.702 INFO:tasks.rados.rados.0.smithi204.stdout:1644: write oid 9 current snap is 0 2024-09-13T11:33:30.702 INFO:tasks.rados.rados.0.smithi204.stdout:1632: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-13T11:33:30.702 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 5719 2024-09-13T11:33:30.702 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 476824 2024-09-13T11:33:30.761 INFO:tasks.rados.rados.0.smithi204.stdout:1635: finishing write tid 1 to smithi20438004-33 2024-09-13T11:33:30.761 INFO:tasks.rados.rados.0.smithi204.stdout:1635: finishing write tid 2 to smithi20438004-33 2024-09-13T11:33:30.761 INFO:tasks.rados.rados.0.smithi204.stdout:1635: finishing write tid 3 to smithi20438004-33 2024-09-13T11:33:30.761 INFO:tasks.rados.rados.0.smithi204.stdout:1635: finishing write tid 4 to smithi20438004-33 2024-09-13T11:33:30.761 INFO:tasks.rados.rados.0.smithi204.stdout:1635: finishing write tid 5 to smithi20438004-33 2024-09-13T11:33:30.761 INFO:tasks.rados.rados.0.smithi204.stdout:1635: oid 33 updating version 0 to 96 2024-09-13T11:33:30.762 INFO:tasks.rados.rados.0.smithi204.stdout:1635: oid 33 updating version 96 to 97 2024-09-13T11:33:30.762 INFO:tasks.rados.rados.0.smithi204.stdout:1635: oid 33 updating version 97 to 98 2024-09-13T11:33:30.762 INFO:tasks.rados.rados.0.smithi204.stdout:1635: oid 33 version 98 is already newer than 95 2024-09-13T11:33:30.762 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 98 (ObjNum 338 snap 0 seq_num 338) dirty exists 2024-09-13T11:33:30.762 INFO:tasks.rados.rados.0.smithi204.stdout:1635: left oid 33 (ObjNum 338 snap 0 seq_num 338) 2024-09-13T11:33:30.762 INFO:tasks.rados.rados.0.smithi204.stdout:1644: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-13T11:33:30.768 INFO:tasks.rados.rados.0.smithi204.stdout:1644: writing smithi20438004-9 from 531187 to 1310628 tid 1 2024-09-13T11:33:30.776 INFO:tasks.rados.rados.0.smithi204.stdout:1644: writing smithi20438004-9 from 2088242 to 2826688 tid 2 2024-09-13T11:33:30.782 INFO:tasks.rados.rados.0.smithi204.stdout:1644: writing smithi20438004-9 from 3431711 to 3731187 tid 3 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1632: done (10 left) 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1635: done (9 left) 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1645: read oid 20 snap -1 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1645: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1646: read oid 24 snap -1 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1646: expect (ObjNum 330 snap 0 seq_num 330) 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1647: read oid 11 snap -1 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1647: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1648: read oid 37 snap -1 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1648: expect deleted 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1649: read oid 31 snap -1 2024-09-13T11:33:30.783 INFO:tasks.rados.rados.0.smithi204.stdout:1649: expect deleted 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:1650: tier_evict oid 15 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:1650: tier_evict oid 15 snap -1 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:1651: set_chunk oid 21 target oid 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:1651: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:1636: finishing set_chunk to oid 4 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 80 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:1638: finishing set_chunk to oid 41 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 41 set_chunk returned error code -95 offset: 489984 length: 1103360 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 0 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-13T11:33:30.784 INFO:tasks.rados.rados.0.smithi204.stdout:1639: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:30.798 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:30.796+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:30.805 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:30.805 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:30.805 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:30.805 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:30.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:30.898 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:30.897+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:30.956 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:30.957 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:30.957 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:30.957 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:30.957 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:30.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:31.000 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:31.000 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:31.000 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:31.000 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:33:31.006 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:31.005+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:1637: got (0) Success 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 0 (ObjNum 310 snap 0 seq_num 310) dirty exists 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:1640: finishing write tid 1 to smithi20438004-14 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:1636: done (15 left) 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:1637: done (14 left) 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:1638: done (13 left) 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:1639: done (12 left) 2024-09-13T11:33:31.030 INFO:tasks.rados.rados.0.smithi204.stdout:1652: write oid 7 current snap is 0 2024-09-13T11:33:31.031 INFO:tasks.rados.rados.0.smithi204.stdout:1640: finishing write tid 2 to smithi20438004-14 2024-09-13T11:33:31.031 INFO:tasks.rados.rados.0.smithi204.stdout:1640: finishing write tid 3 to smithi20438004-14 2024-09-13T11:33:31.031 INFO:tasks.rados.rados.0.smithi204.stdout:1652: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-13T11:33:31.035 INFO:tasks.rados.rados.0.smithi204.stdout:1652: writing smithi20438004-7 from 547994 to 1002684 tid 1 2024-09-13T11:33:31.042 INFO:tasks.rados.rados.0.smithi204.stdout:1652: writing smithi20438004-7 from 1746724 to 2228828 tid 2 2024-09-13T11:33:31.047 INFO:tasks.rados.rados.0.smithi204.stdout:1652: writing smithi20438004-7 from 2659134 to 3124280 tid 3 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1652: writing smithi20438004-7 from 3605976 to 3747994 tid 4 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1653: read oid 48 snap -1 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1653: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1654: write oid 5 current snap is 0 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1640: finishing write tid 4 to smithi20438004-14 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1640: finishing write tid 5 to smithi20438004-14 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1640: oid 14 updating version 0 to 121 2024-09-13T11:33:31.052 INFO:tasks.rados.rados.0.smithi204.stdout:1640: oid 14 updating version 121 to 122 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1640: oid 14 version 122 is already newer than 120 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1640: oid 14 version 122 is already newer than 119 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 122 (ObjNum 339 snap 0 seq_num 339) dirty exists 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1640: left oid 14 (ObjNum 339 snap 0 seq_num 339) 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1641: finishing set_chunk to oid 49 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 1803776 length: 471552 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1642: finishing write tid 1 to smithi20438004-34 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1642: finishing write tid 2 to smithi20438004-34 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1642: finishing write tid 3 to smithi20438004-34 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1642: finishing write tid 4 to smithi20438004-34 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1642: finishing write tid 5 to smithi20438004-34 2024-09-13T11:33:31.053 INFO:tasks.rados.rados.0.smithi204.stdout:1642: oid 34 updating version 0 to 133 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1642: oid 34 updating version 133 to 134 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1642: oid 34 version 134 is already newer than 131 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1642: oid 34 version 134 is already newer than 132 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 134 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1642: left oid 34 (ObjNum 340 snap 0 seq_num 340) 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1643: got (0) Success 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 86 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1644: finishing write tid 1 to smithi20438004-9 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1644: finishing write tid 2 to smithi20438004-9 2024-09-13T11:33:31.054 INFO:tasks.rados.rados.0.smithi204.stdout:1654: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-13T11:33:31.059 INFO:tasks.rados.rados.0.smithi204.stdout:1654: writing smithi20438004-5 from 564801 to 1211093 tid 1 2024-09-13T11:33:31.066 INFO:tasks.rados.rados.0.smithi204.stdout:1654: writing smithi20438004-5 from 1921559 to 2663674 tid 2 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1654: writing smithi20438004-5 from 3202910 to 3764801 tid 3 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1640: done (14 left) 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1641: done (13 left) 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1642: done (12 left) 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1643: done (11 left) 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1655: read oid 38 snap -1 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1655: expect (ObjNum 337 snap 0 seq_num 337) 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1650: got (22) Invalid argument 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1651: finishing set_chunk to oid 21 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1651: got expected ENOENT (src dne) 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1656: read oid 30 snap -1 2024-09-13T11:33:31.073 INFO:tasks.rados.rados.0.smithi204.stdout:1656: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:1648: done (12 left) 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:1649: done (11 left) 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:1650: done (10 left) 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:1651: done (9 left) 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:1657: tier_promote oid 41 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:1644: finishing write tid 3 to smithi20438004-9 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:1647: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:31.074 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:31.100 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:31.099+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:31.110 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 86.735548980999994, 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27446322899999998, 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1100000000000005e-07 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372244+0000", 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.66e-06 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372256+0000", 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1817000000000001e-05 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372264+0000", 2024-09-13T11:33:31.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0220000000000004e-06 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372320+0000", 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6437999999999997e-05 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372329+0000", 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644121+0000", 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27179224899999999 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644271+0000", 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014990000000000001 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644421+0000", 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015035600000000001 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646446+0000", 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00202454 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646524+0000", 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8839999999999997e-05 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646598+0000", 2024-09-13T11:33:31.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3862999999999994e-05 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646654+0000", 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5609999999999998e-05 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646701+0000", 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6844000000000001e-05 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:207 3.2 3:59a396eb:::smithi20438004-8 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.023151933999998, 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063519737000000007, 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 207 2024-09-13T11:33:31.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7099999999999998e-07 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084636+0000", 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.559e-06 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084640+0000", 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.118e-06 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084644+0000", 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8640000000000004e-06 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084687+0000", 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2295000000000001e-05 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084780+0000", 2024-09-13T11:33:31.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2880000000000005e-05 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.106902+0000", 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022122822 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112195+0000", 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052927 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.121161+0000", 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089661740000000004 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148052+0000", 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026890595 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148105+0000", 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2673000000000002e-05 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148154+0000", 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9786000000000001e-05 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:208 3.2 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.023133217999998, 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049409478, 2024-09-13T11:33:31.124 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 208 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084654+0000", 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084657+0000", 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1769999999999998e-06 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084663+0000", 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8329999999999999e-06 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084668+0000", 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6200000000000004e-06 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084884+0000", 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021599499999999999 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.085046+0000", 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016192600000000001 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.094613+0000", 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095668200000000002 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.101900+0000", 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072866160000000001 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.110249+0000", 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008349034 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133903+0000", 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023654272 2024-09-13T11:33:31.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133976+0000", 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2581000000000002e-05 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134063+0000", 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7169000000000005e-05 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:211 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.023092188999996, 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047621726000000003, 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 211 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084695+0000", 2024-09-13T11:33:31.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6100000000000002e-07 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084697+0000", 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2419999999999999e-06 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8160000000000001e-06 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084706+0000", 2024-09-13T11:33:31.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.313e-06 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086622+0000", 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001916212 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086720+0000", 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8077000000000003e-05 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100598+0000", 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013877195 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104958+0000", 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043607899999999998 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115608+0000", 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010650138 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132197+0000", 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016589008999999998 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132257+0000", 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9948000000000003e-05 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132316+0000", 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8625000000000002e-05 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.136 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:212 3.3 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.023074609000005, 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063179242999999996, 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 212 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7300000000000002e-07 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:31.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.255e-06 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084719+0000", 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4800000000000003e-06 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084724+0000", 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.527e-06 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086839+0000", 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021155240000000001 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086947+0000", 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107675 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109128+0000", 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022180947999999999 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.113055+0000", 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039269029999999998 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.119148+0000", 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060933810000000001 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147786+0000", 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028638395000000001 2024-09-13T11:33:31.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147838+0000", 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1405e-05 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147891+0000", 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3377000000000001e-05 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:216 3.2 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.017135362000005, 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055588580999999998, 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 216 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.144 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090653+0000", 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090663+0000", 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0708e-05 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090675+0000", 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090688+0000", 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3193000000000001e-05 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096478+0000", 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057898150000000002 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096694+0000", 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021582700000000001 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.103295+0000", 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006601048 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108559+0000", 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052645449999999998 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.114944+0000", 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063845389999999998 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146121+0000", 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031177449999999999 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146186+0000", 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5235999999999997e-05 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146240+0000", 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3368000000000001e-05 2024-09-13T11:33:31.149 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:220 3.2 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.991959838, 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033838506999999997, 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.150 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 220 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115828+0000", 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115832+0000", 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.746e-06 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115842+0000", 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2010000000000005e-06 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115847+0000", 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9740000000000001e-06 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115897+0000", 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0114999999999997e-05 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116103+0000", 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020598799999999999 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135329+0000", 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019226422999999999 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141061+0000", 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057316709999999998 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.142490+0000", 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001428805 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149552+0000", 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070625610000000002 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149602+0000", 2024-09-13T11:33:31.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9826e-05 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149665+0000", 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3168000000000002e-05 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:226 3.3 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.973408391, 2024-09-13T11:33:31.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.062274747999999998, 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 226 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:31.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134379+0000", 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1850000000000001e-06 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134392+0000", 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2649999999999999e-05 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134404+0000", 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1615e-05 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134411+0000", 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3509999999999996e-06 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134474+0000", 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2673000000000001e-05 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134662+0000", 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018810100000000001 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149047+0000", 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014384902 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.154919+0000", 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058721379999999998 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.171898+0000", 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016979300999999999 2024-09-13T11:33:31.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196539+0000", 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024641252999999998 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196614+0000", 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5160999999999994e-05 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196653+0000", 2024-09-13T11:33:31.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8417999999999997e-05 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:228 3.3 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.967125858999999, 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038699092999999997, 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.163 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140662+0000", 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140670+0000", 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8630000000000008e-06 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140696+0000", 2024-09-13T11:33:31.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5927000000000001e-05 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140708+0000", 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140804+0000", 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6137999999999993e-05 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140995+0000", 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191316 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.145999+0000", 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050033129999999997 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152237+0000", 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062383320000000001 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160262+0000", 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080251520000000007 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179164+0000", 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018901823000000002 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179296+0000", 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013243400000000001 2024-09-13T11:33:31.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179360+0000", 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3328000000000001e-05 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:229 3.3 3:d73fa370:::smithi20438004-30 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.966991831999998, 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057728689, 2024-09-13T11:33:31.169 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 229 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140796+0000", 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9599999999999998e-07 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140800+0000", 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:33:31.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140809+0000", 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3669999999999997e-06 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140814+0000", 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141203+0000", 2024-09-13T11:33:31.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038959299999999998 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141375+0000", 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017155499999999999 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.156365+0000", 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014989954999999999 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162876+0000", 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065109360000000002 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.187965+0000", 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025088847000000001 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198445+0000", 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010480171 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198487+0000", 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2635000000000001e-05 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198523+0000", 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5941e-05 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:238 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:31.175 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.955026857, 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.059471032, 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 238 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:31.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152761+0000", 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152772+0000", 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1122e-05 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152782+0000", 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5179999999999993e-06 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152790+0000", 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7260000000000002e-06 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152856+0000", 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6304999999999996e-05 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153180+0000", 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000323617 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167201+0000", 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014021264 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.172985+0000", 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057843349999999998 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188702+0000", 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015716765000000001 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212106+0000", 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023403714999999999 2024-09-13T11:33:31.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212169+0000", 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3224000000000003e-05 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212231+0000", 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1841999999999995e-05 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:240 3.2 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.947536880999998, 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034186595, 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.182 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160251+0000", 2024-09-13T11:33:31.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.562e-06 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160258+0000", 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6640000000000001e-06 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160269+0000", 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.152e-05 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160277+0000", 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4939999999999997e-06 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160327+0000", 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0268000000000001e-05 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160497+0000", 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017018900000000001 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.164999+0000", 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045015929999999999 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168431+0000", 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003431698 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.184172+0000", 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015741852000000001 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194334+0000", 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010161646999999999 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194390+0000", 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6218999999999998e-05 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194436+0000", 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5889000000000001e-05 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.187 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:242 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 18] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.939149073999999, 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053252359999999999, 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.188 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 242 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168640+0000", 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168652+0000", 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1909000000000001e-05 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168668+0000", 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5888000000000001e-05 2024-09-13T11:33:31.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168680+0000", 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.154e-05 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169643+0000", 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096305100000000003 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:31.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000139834 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174850+0000", 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005067841 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.178949+0000", 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040991209999999998 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193069+0000", 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014119166000000001 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221801+0000", 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028732905999999999 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221852+0000", 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0958000000000002e-05 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221890+0000", 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7398999999999999e-05 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:244 3.2 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.927209050000002, 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040499786000000003, 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.194 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180579+0000", 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.499e-06 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180585+0000", 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4649999999999999e-06 2024-09-13T11:33:31.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180598+0000", 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180604+0000", 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5300000000000004e-06 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180669+0000", 2024-09-13T11:33:31.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4859999999999999e-05 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180840+0000", 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171791 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.190029+0000", 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091888999999999998 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193312+0000", 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829579999999999 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.207773+0000", 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014461039 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.220989+0000", 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013215644 2024-09-13T11:33:31.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221038+0000", 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9421e-05 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221077+0000", 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8899999999999997e-05 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.200 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:247 3.2 3:4f19c0b8:::smithi20438004-48 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.921972146000002, 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040924556000000001, 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.201 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 247 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185816+0000", 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9690000000000001e-06 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185825+0000", 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6980000000000004e-06 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185842+0000", 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6492000000000001e-05 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185852+0000", 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0412e-05 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185951+0000", 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9312000000000001e-05 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186223+0000", 2024-09-13T11:33:31.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027177000000000001 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208403+0000", 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022179780999999999 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029604150000000001 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.223543+0000", 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012179840000000001 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226659+0000", 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031159989999999999 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226698+0000", 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9020999999999998e-05 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226739+0000", 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0846999999999997e-05 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:248 3.2 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.921649064999997, 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035109013000000001, 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.207 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 248 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186139+0000", 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.333e-06 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186157+0000", 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8488999999999999e-05 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186170+0000", 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2737e-05 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186181+0000", 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1279e-05 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186356+0000", 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174988 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186493+0000", 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137045 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.203390+0000", 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016897169 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.206673+0000", 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829510000000001 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.214284+0000", 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076100830000000001 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221141+0000", 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068573740000000003 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221198+0000", 2024-09-13T11:33:31.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6916999999999997e-05 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221246+0000", 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8647999999999999e-05 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.10.22:30 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.316611989000002, 2024-09-13T11:33:31.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87173941200000005, 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791176+0000", 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791183+0000", 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.956e-06 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791193+0000", 2024-09-13T11:33:31.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8649999999999996e-06 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791200+0000", 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.148e-06 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791263+0000", 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3038000000000002e-05 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791370+0000", 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010721999999999999 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791704+0000", 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033406100000000003 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.792618+0000", 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091326999999999999 2024-09-13T11:33:31.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.793087+0000", 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046985199999999999 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662893+0000", 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86980523300000001 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662914+0000", 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1319e-05 2024-09-13T11:33:31.218 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.1.22:76 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.302964476, 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85814986199999999, 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:31.219 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804824+0000", 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5889999999999999e-06 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804830+0000", 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7139999999999996e-06 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804845+0000", 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5158e-05 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804853+0000", 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9349999999999994e-06 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804910+0000", 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6501e-05 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805017+0000", 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107237 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805213+0000", 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019558100000000001 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806246+0000", 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001033225 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806260+0000", 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4072e-05 2024-09-13T11:33:31.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662951+0000", 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85669131700000001 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662972+0000", 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0533000000000001e-05 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.224 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:9 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.262440775000002, 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81776348899999995, 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845348+0000", 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4090000000000001e-06 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845360+0000", 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1420000000000001e-05 2024-09-13T11:33:31.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845389+0000", 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9453000000000001e-05 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845404+0000", 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4919000000000001e-05 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845526+0000", 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121588 2024-09-13T11:33:31.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845617+0000", 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1119000000000006e-05 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845789+0000", 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017257000000000001 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846396+0000", 2024-09-13T11:33:31.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060634700000000003 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846771+0000", 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037541099999999997 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663066+0000", 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81629477399999995 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663109+0000", 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3479000000000002e-05 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:10 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.258704346000002, 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81407563599999999, 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:31.230 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849083+0000", 2024-09-13T11:33:31.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6300000000000004e-07 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849089+0000", 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849101+0000", 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-05 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:31.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849106+0000", 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7309999999999998e-06 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849154+0000", 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8077000000000001e-05 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849235+0000", 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0770000000000001e-05 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:31.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849399+0000", 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163385 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850247+0000", 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084889399999999999 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850266+0000", 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8698e-05 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663146+0000", 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81287977300000003 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663158+0000", 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1928e-05 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:31.235 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:31.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:31.237 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:31.238 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:31.236+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:31.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:31.304 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2030766 2024-09-13T11:33:31.314 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:33:31.314 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1644: finishing write tid 4 to smithi20438004-9 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1644: finishing write tid 5 to smithi20438004-9 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1644: oid 9 updating version 0 to 55 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1644: oid 9 version 55 is already newer than 53 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1644: oid 9 version 55 is already newer than 52 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1644: oid 9 version 55 is already newer than 54 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 55 (ObjNum 341 snap 0 seq_num 341) dirty exists 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1644: left oid 9 (ObjNum 341 snap 0 seq_num 341) 2024-09-13T11:33:31.323 INFO:tasks.rados.rados.0.smithi204.stdout:1645: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:31.326 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-13T11:33:30.944716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":204670124,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":79,"num_objects_misplaced":56,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":3572,"num_read_kb":2156985,"num_write":2486,"num_write_kb":570288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":112279409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":98,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":839216,"kb_used_data":494128,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1131622864,"statfs":{"total":1159641169920,"available":1158781812736,"internally_reserved":0,"allocated":505987072,"data_stored":500792057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19137,"internal_metadata":353154367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2804557,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":293,"num_read_kb":237523,"num_write":130,"num_write_kb":36089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.630717"},"pg_stats":[{"pgid":"4.19","version":"53'20","reported_seq":185,"reported_epoch":77,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:26.642600+0000","last_change":"2024-09-13T11:33:12.028705+0000","last_active":"2024-09-13T11:33:26.642600+0000","last_peered":"2024-09-13T11:33:26.642600+0000","last_clean":"2024-09-13T11:33:26.642600+0000","last_became_active":"2024-09-13T11:32:59.281878+0000","last_became_peered":"2024-09-13T11:32:59.281878+0000","last_unstale":"2024-09-13T11:33:26.642600+0000","last_undegraded":"2024-09-13T11:33:26.642600+0000","last_fullsized":"2024-09-13T11:33:26.642600+0000","mapping_epoch":61,"log_start":"51'17","ondisk_log_start":"51'17","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'20","last_scrub_stamp":"2024-09-13T11:33:12.028588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.028588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:17.408780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6,0,7],"avail_no_missing":["6","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"76'78","reported_seq":297,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.485064+0000","last_change":"2024-09-13T11:33:16.278769+0000","last_active":"2024-09-13T11:33:26.485064+0000","last_peered":"2024-09-13T11:33:26.485064+0000","last_clean":"2024-09-13T11:33:26.485064+0000","last_became_active":"2024-09-13T11:33:16.278471+0000","last_became_peered":"2024-09-13T11:33:16.278471+0000","last_unstale":"2024-09-13T11:33:26.485064+0000","last_undegraded":"2024-09-13T11:33:26.485064+0000","last_fullsized":"2024-09-13T11:33:26.485064+0000","mapping_epoch":72,"log_start":"76'75","ondisk_log_start":"76'75","created":56,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'57","last_scrub_stamp":"2024-09-13T11:33:04.555283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.555283+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":73,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:32.651867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5815288,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":55,"num_read_kb":44799,"num_write":26,"num_write_kb":6447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,6],"acting":[11,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":59,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.315596+0000","last_change":"2024-09-13T11:33:03.196131+0000","last_active":"2024-09-13T11:33:26.315596+0000","last_peered":"2024-09-13T11:33:26.315596+0000","last_clean":"2024-09-13T11:33:26.315596+0000","last_became_active":"2024-09-13T11:32:56.278133+0000","last_became_peered":"2024-09-13T11:32:56.278133+0000","last_unstale":"2024-09-13T11:33:26.315596+0000","last_undegraded":"2024-09-13T11:33:26.315596+0000","last_fullsized":"2024-09-13T11:33:26.315596+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.196058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:32.709794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6],"acting":[14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.16","version":"75'89","reported_seq":326,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:26.642703+0000","last_change":"2024-09-13T11:32:56.273159+0000","last_active":"2024-09-13T11:33:26.642703+0000","last_peered":"2024-09-13T11:33:26.642703+0000","last_clean":"2024-09-13T11:32:55.237210+0000","last_became_active":"2024-09-13T11:32:56.273159+0000","last_became_peered":"2024-09-13T11:32:56.273159+0000","last_unstale":"2024-09-13T11:33:26.642703+0000","last_undegraded":"2024-09-13T11:32:56.268297+0000","last_fullsized":"2024-09-13T11:32:56.268083+0000","mapping_epoch":58,"log_start":"74'86","ondisk_log_start":"74'86","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":85,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2092488,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":185,"num_read_kb":177096,"num_write":62,"num_write_kb":19589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":1}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"60'41","reported_seq":231,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.642232+0000","last_change":"2024-09-13T11:33:05.991060+0000","last_active":"2024-09-13T11:33:26.642232+0000","last_peered":"2024-09-13T11:33:26.642232+0000","last_clean":"2024-09-13T11:33:26.642232+0000","last_became_active":"2024-09-13T11:32:59.285266+0000","last_became_peered":"2024-09-13T11:32:59.285266+0000","last_unstale":"2024-09-13T11:33:26.642232+0000","last_undegraded":"2024-09-13T11:33:26.642232+0000","last_fullsized":"2024-09-13T11:33:26.642232+0000","mapping_epoch":61,"log_start":"52'36","ondisk_log_start":"52'36","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'41","last_scrub_stamp":"2024-09-13T11:33:05.990981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.990981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":36,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:30.628856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,6],"acting":[0,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"76'103","reported_seq":373,"reported_epoch":77,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:26.315144+0000","last_change":"2024-09-13T11:33:04.630617+0000","last_active":"2024-09-13T11:33:26.315144+0000","last_peered":"2024-09-13T11:33:26.315144+0000","last_clean":"2024-09-13T11:33:26.315144+0000","last_became_active":"2024-09-13T11:32:58.287813+0000","last_became_peered":"2024-09-13T11:32:58.287813+0000","last_unstale":"2024-09-13T11:33:26.315144+0000","last_undegraded":"2024-09-13T11:33:26.315144+0000","last_fullsized":"2024-09-13T11:33:26.315144+0000","mapping_epoch":60,"log_start":"76'100","ondisk_log_start":"76'100","created":56,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'78","last_scrub_stamp":"2024-09-13T11:33:04.630512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.630512+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":100,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:28.662873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9647317,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":96,"num_read_kb":73478,"num_write":28,"num_write_kb":7807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,3],"avail_no_missing":["4","3","7"],"object_location_counts":[{"shards":"3,4,7","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"53'41","reported_seq":216,"reported_epoch":79,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:28.412088+0000","last_change":"2024-09-13T11:33:05.531596+0000","last_active":"2024-09-13T11:33:28.412088+0000","last_peered":"2024-09-13T11:33:28.412088+0000","last_clean":"2024-09-13T11:33:28.412088+0000","last_became_active":"2024-09-13T11:32:59.285096+0000","last_became_peered":"2024-09-13T11:32:59.285096+0000","last_unstale":"2024-09-13T11:33:28.412088+0000","last_undegraded":"2024-09-13T11:33:28.412088+0000","last_fullsized":"2024-09-13T11:33:28.412088+0000","mapping_epoch":61,"log_start":"50'38","ondisk_log_start":"50'38","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'41","last_scrub_stamp":"2024-09-13T11:33:05.531507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.531507+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:13.447752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,14],"acting":[7,14,6],"avail_no_missing":["7","6","14"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"76'114","reported_seq":390,"reported_epoch":77,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:26.315569+0000","last_change":"2024-09-13T11:33:05.196910+0000","last_active":"2024-09-13T11:33:26.315569+0000","last_peered":"2024-09-13T11:33:26.315569+0000","last_clean":"2024-09-13T11:33:26.315569+0000","last_became_active":"2024-09-13T11:32:59.288348+0000","last_became_peered":"2024-09-13T11:32:59.288348+0000","last_unstale":"2024-09-13T11:33:26.315569+0000","last_undegraded":"2024-09-13T11:33:26.315569+0000","last_fullsized":"2024-09-13T11:33:26.315569+0000","mapping_epoch":61,"log_start":"76'111","ondisk_log_start":"76'111","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'93","last_scrub_stamp":"2024-09-13T11:33:05.196778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.196778+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":111,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:09.913125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9210335,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":61782,"num_write":25,"num_write_kb":7216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8],"acting":[14,8,3],"avail_no_missing":["14","3","8"],"object_location_counts":[{"shards":"3,8,14","objects":3}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"76'129","reported_seq":459,"reported_epoch":79,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:29.084125+0000","last_change":"2024-09-13T11:33:03.598922+0000","last_active":"2024-09-13T11:33:29.084125+0000","last_peered":"2024-09-13T11:33:29.084125+0000","last_clean":"2024-09-13T11:33:29.084125+0000","last_became_active":"2024-09-13T11:32:57.287130+0000","last_became_peered":"2024-09-13T11:32:57.287130+0000","last_unstale":"2024-09-13T11:33:29.084125+0000","last_undegraded":"2024-09-13T11:33:29.084125+0000","last_fullsized":"2024-09-13T11:33:29.084125+0000","mapping_epoch":59,"log_start":"76'126","ondisk_log_start":"76'126","created":56,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'103","last_scrub_stamp":"2024-09-13T11:33:03.598827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.598827+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:29.147515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10101040,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":141,"num_read_kb":157439,"num_write":26,"num_write_kb":9625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,15],"avail_no_missing":["7","3","15"],"object_location_counts":[{"shards":"3,7,15","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"76'68","reported_seq":292,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.642698+0000","last_change":"2024-09-13T11:33:12.208901+0000","last_active":"2024-09-13T11:33:26.642698+0000","last_peered":"2024-09-13T11:33:26.642698+0000","last_clean":"2024-09-13T11:33:26.642698+0000","last_became_active":"2024-09-13T11:32:59.280441+0000","last_became_peered":"2024-09-13T11:32:59.280441+0000","last_unstale":"2024-09-13T11:33:26.642698+0000","last_undegraded":"2024-09-13T11:33:26.642698+0000","last_fullsized":"2024-09-13T11:33:26.642698+0000","mapping_epoch":61,"log_start":"76'65","ondisk_log_start":"76'65","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'60","last_scrub_stamp":"2024-09-13T11:33:12.208834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.208834+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:21.081399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":4794,"num_write":10,"num_write_kb":2746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,3],"acting":[10,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":129,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T11:33:28.411629+0000","last_change":"2024-09-13T11:32:45.278444+0000","last_active":"2024-09-13T11:33:28.411629+0000","last_peered":"2024-09-13T11:33:28.411629+0000","last_clean":"2024-09-13T11:33:28.411629+0000","last_became_active":"2024-09-13T11:32:45.277643+0000","last_became_peered":"2024-09-13T11:32:45.277643+0000","last_unstale":"2024-09-13T11:33:28.411629+0000","last_undegraded":"2024-09-13T11:33:28.411629+0000","last_fullsized":"2024-09-13T11:33:28.411629+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,7],"acting":[15,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.1","version":"79'122","reported_seq":411,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.144608+0000","last_change":"2024-09-13T11:33:01.310446+0000","last_active":"2024-09-13T11:33:29.144608+0000","last_peered":"2024-09-13T11:33:29.144608+0000","last_clean":"2024-09-13T11:33:29.144608+0000","last_became_active":"2024-09-13T11:32:54.604852+0000","last_became_peered":"2024-09-13T11:32:54.604852+0000","last_unstale":"2024-09-13T11:33:29.144608+0000","last_undegraded":"2024-09-13T11:33:29.144608+0000","last_fullsized":"2024-09-13T11:33:29.144608+0000","mapping_epoch":52,"log_start":"75'119","ondisk_log_start":"75'119","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'105","last_scrub_stamp":"2024-09-13T11:33:01.310380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.310380+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":117,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:08.141302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5277464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":28629,"num_write":16,"num_write_kb":6395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3,7],"acting":[15,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":119,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.315664+0000","last_change":"2024-09-13T11:32:08.625099+0000","last_active":"2024-09-13T11:33:26.315664+0000","last_peered":"2024-09-13T11:33:26.315664+0000","last_clean":"2024-09-13T11:33:26.315664+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:33:26.315664+0000","last_undegraded":"2024-09-13T11:33:26.315664+0000","last_fullsized":"2024-09-13T11:33:26.315664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:31.093378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"75'78","reported_seq":304,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.642727+0000","last_change":"2024-09-13T11:33:02.247329+0000","last_active":"2024-09-13T11:33:26.642727+0000","last_peered":"2024-09-13T11:33:26.642727+0000","last_clean":"2024-09-13T11:33:26.642727+0000","last_became_active":"2024-09-13T11:32:54.604560+0000","last_became_peered":"2024-09-13T11:32:54.604560+0000","last_unstale":"2024-09-13T11:33:26.642727+0000","last_undegraded":"2024-09-13T11:33:26.642727+0000","last_fullsized":"2024-09-13T11:33:26.642727+0000","mapping_epoch":51,"log_start":"74'75","ondisk_log_start":"74'75","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'57","last_scrub_stamp":"2024-09-13T11:33:02.247255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.247255+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:19.872698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647218,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":64016,"num_write":26,"num_write_kb":7849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":123,"reported_epoch":77,"state":"active+undersized","last_fresh":"2024-09-13T11:33:26.642651+0000","last_change":"2024-09-13T11:32:32.498855+0000","last_active":"2024-09-13T11:33:26.642651+0000","last_peered":"2024-09-13T11:33:26.642651+0000","last_clean":"2024-09-13T11:32:30.479480+0000","last_became_active":"2024-09-13T11:32:32.498855+0000","last_became_peered":"2024-09-13T11:32:32.498855+0000","last_unstale":"2024-09-13T11:33:26.642651+0000","last_undegraded":"2024-09-13T11:33:26.642651+0000","last_fullsized":"2024-09-13T11:32:32.497672+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:35.325284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":["8","6"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"76'47","reported_seq":207,"reported_epoch":77,"state":"active+undersized","last_fresh":"2024-09-13T11:33:26.485108+0000","last_change":"2024-09-13T11:33:26.485108+0000","last_active":"2024-09-13T11:33:26.485108+0000","last_peered":"2024-09-13T11:33:26.485108+0000","last_clean":"2024-09-13T11:32:49.952102+0000","last_became_active":"2024-09-13T11:32:57.283497+0000","last_became_peered":"2024-09-13T11:32:57.283497+0000","last_unstale":"2024-09-13T11:33:26.485108+0000","last_undegraded":"2024-09-13T11:33:26.485108+0000","last_fullsized":"2024-09-13T11:32:57.277969+0000","mapping_epoch":59,"log_start":"70'44","ondisk_log_start":"70'44","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":44,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:26.973844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":31111,"num_write":26,"num_write_kb":8121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2383182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":["11","6"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":122,"reported_epoch":77,"state":"active+undersized","last_fresh":"2024-09-13T11:33:26.642467+0000","last_change":"2024-09-13T11:32:57.283324+0000","last_active":"2024-09-13T11:33:26.642467+0000","last_peered":"2024-09-13T11:33:26.642467+0000","last_clean":"2024-09-13T11:32:55.276320+0000","last_became_active":"2024-09-13T11:32:57.283324+0000","last_became_peered":"2024-09-13T11:32:57.283324+0000","last_unstale":"2024-09-13T11:33:26.642467+0000","last_undegraded":"2024-09-13T11:33:26.642467+0000","last_fullsized":"2024-09-13T11:32:57.273798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.914763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"76'130","reported_seq":378,"reported_epoch":77,"state":"active+undersized+remapped","last_fresh":"2024-09-13T11:33:26.332153+0000","last_change":"2024-09-13T11:32:57.284225+0000","last_active":"2024-09-13T11:33:26.332153+0000","last_peered":"2024-09-13T11:33:26.332153+0000","last_clean":"2024-09-13T11:32:55.273515+0000","last_became_active":"2024-09-13T11:32:57.284225+0000","last_became_peered":"2024-09-13T11:32:57.284225+0000","last_unstale":"2024-09-13T11:33:26.332153+0000","last_undegraded":"2024-09-13T11:33:26.332153+0000","last_fullsized":"2024-09-13T11:32:57.278637+0000","mapping_epoch":59,"log_start":"76'127","ondisk_log_start":"76'127","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:29.755429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":116460,"num_write":96,"num_write_kb":28168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4501279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3,9],"avail_no_missing":["3","9"],"object_location_counts":[{"shards":"3,9","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"76'252","reported_seq":436,"reported_epoch":79,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:28.412121+0000","last_change":"2024-09-13T11:32:48.283369+0000","last_active":"2024-09-13T11:33:28.412121+0000","last_peered":"2024-09-13T11:33:28.412121+0000","last_clean":"2024-09-13T11:33:28.412121+0000","last_became_active":"2024-09-13T11:32:48.281772+0000","last_became_peered":"2024-09-13T11:32:48.281772+0000","last_unstale":"2024-09-13T11:33:28.412121+0000","last_undegraded":"2024-09-13T11:33:28.412121+0000","last_fullsized":"2024-09-13T11:33:28.412121+0000","mapping_epoch":52,"log_start":"76'249","ondisk_log_start":"76'249","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":258,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22992714,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":226,"num_read_kb":5985,"num_write":331,"num_write_kb":65951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":14112815,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15],"acting":[7,15,13],"avail_no_missing":["7","13","15"],"object_location_counts":[{"shards":"7,13,15","objects":42}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":139,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.770230+0000","last_change":"2024-09-13T11:33:26.770230+0000","last_active":"2024-09-13T11:33:26.770230+0000","last_peered":"2024-09-13T11:33:26.770230+0000","last_clean":"2024-09-13T11:33:26.770230+0000","last_became_active":"2024-09-13T11:32:43.554077+0000","last_became_peered":"2024-09-13T11:32:43.554077+0000","last_unstale":"2024-09-13T11:33:26.770230+0000","last_undegraded":"2024-09-13T11:33:26.770230+0000","last_fullsized":"2024-09-13T11:33:26.770230+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:33:26.770137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:26.770137+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:51.278083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,11],"acting":[3,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"75'81","reported_seq":318,"reported_epoch":77,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:26.332096+0000","last_change":"2024-09-13T11:33:01.558907+0000","last_active":"2024-09-13T11:33:26.332096+0000","last_peered":"2024-09-13T11:33:26.332096+0000","last_clean":"2024-09-13T11:33:26.332096+0000","last_became_active":"2024-09-13T11:32:54.600364+0000","last_became_peered":"2024-09-13T11:32:54.600364+0000","last_unstale":"2024-09-13T11:33:26.332096+0000","last_undegraded":"2024-09-13T11:33:26.332096+0000","last_fullsized":"2024-09-13T11:33:26.332096+0000","mapping_epoch":49,"log_start":"74'78","ondisk_log_start":"74'78","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'73","last_scrub_stamp":"2024-09-13T11:33:01.558809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.558809+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:23.044953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857223,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":45061,"num_write":9,"num_write_kb":2663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8,5],"avail_no_missing":["3","5","8"],"object_location_counts":[{"shards":"3,5,8","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"77'259","reported_seq":453,"reported_epoch":79,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:28.411562+0000","last_change":"2024-09-13T11:32:57.280350+0000","last_active":"2024-09-13T11:33:28.411562+0000","last_peered":"2024-09-13T11:33:28.411562+0000","last_clean":"2024-09-13T11:32:54.594742+0000","last_became_active":"2024-09-13T11:32:57.280350+0000","last_became_peered":"2024-09-13T11:32:57.280350+0000","last_unstale":"2024-09-13T11:33:28.411562+0000","last_undegraded":"2024-09-13T11:32:57.275975+0000","last_fullsized":"2024-09-13T11:32:57.275775+0000","mapping_epoch":59,"log_start":"76'256","ondisk_log_start":"76'256","created":30,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":262,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:32.824670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26548115,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":237,"num_read_kb":11674,"num_write":336,"num_write_kb":73321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":24388633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6],"acting":[15,6],"avail_no_missing":["15","6"],"object_location_counts":[{"shards":"6,15","objects":37}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":123,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.315593+0000","last_change":"2024-09-13T11:32:45.284485+0000","last_active":"2024-09-13T11:33:26.315593+0000","last_peered":"2024-09-13T11:33:26.315593+0000","last_clean":"2024-09-13T11:33:26.315593+0000","last_became_active":"2024-09-13T11:32:45.284134+0000","last_became_peered":"2024-09-13T11:32:45.284134+0000","last_unstale":"2024-09-13T11:33:26.315593+0000","last_undegraded":"2024-09-13T11:33:26.315593+0000","last_fullsized":"2024-09-13T11:33:26.315593+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,8],"acting":[14,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":135,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.642331+0000","last_change":"2024-09-13T11:33:03.236076+0000","last_active":"2024-09-13T11:33:26.642331+0000","last_peered":"2024-09-13T11:33:26.642331+0000","last_clean":"2024-09-13T11:33:26.642331+0000","last_became_active":"2024-09-13T11:32:54.601517+0000","last_became_peered":"2024-09-13T11:32:54.601517+0000","last_unstale":"2024-09-13T11:33:26.642331+0000","last_undegraded":"2024-09-13T11:33:26.642331+0000","last_fullsized":"2024-09-13T11:33:26.642331+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.236001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.236001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:06.939201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"76'299","reported_seq":505,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.642567+0000","last_change":"2024-09-13T11:32:47.277934+0000","last_active":"2024-09-13T11:33:26.642567+0000","last_peered":"2024-09-13T11:33:26.642567+0000","last_clean":"2024-09-13T11:33:26.642567+0000","last_became_active":"2024-09-13T11:32:47.277392+0000","last_became_peered":"2024-09-13T11:32:47.277392+0000","last_unstale":"2024-09-13T11:33:26.642567+0000","last_undegraded":"2024-09-13T11:33:26.642567+0000","last_fullsized":"2024-09-13T11:33:26.642567+0000","mapping_epoch":51,"log_start":"76'296","ondisk_log_start":"76'296","created":30,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":311,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:40.554823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23723612,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":259,"num_read_kb":4471,"num_write":370,"num_write_kb":82679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":28975184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,0],"acting":[6,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":118,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.332152+0000","last_change":"2024-09-13T11:32:08.377316+0000","last_active":"2024-09-13T11:33:26.332152+0000","last_peered":"2024-09-13T11:33:26.332152+0000","last_clean":"2024-09-13T11:33:26.332152+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:33:26.332152+0000","last_undegraded":"2024-09-13T11:33:26.332152+0000","last_fullsized":"2024-09-13T11:33:26.332152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:26.510810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"75'82","reported_seq":318,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:26.642825+0000","last_change":"2024-09-13T11:32:57.280636+0000","last_active":"2024-09-13T11:33:26.642825+0000","last_peered":"2024-09-13T11:33:26.642825+0000","last_clean":"2024-09-13T11:32:54.904504+0000","last_became_active":"2024-09-13T11:32:57.280636+0000","last_became_peered":"2024-09-13T11:32:57.280636+0000","last_unstale":"2024-09-13T11:33:26.642825+0000","last_undegraded":"2024-09-13T11:32:57.277736+0000","last_fullsized":"2024-09-13T11:32:57.277594+0000","mapping_epoch":59,"log_start":"74'79","ondisk_log_start":"74'79","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":77,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:27.139801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":175,"num_read_kb":122611,"num_write":51,"num_write_kb":17057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"77'318","reported_seq":504,"reported_epoch":79,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:28.411454+0000","last_change":"2024-09-13T11:33:16.277928+0000","last_active":"2024-09-13T11:33:28.411454+0000","last_peered":"2024-09-13T11:33:28.411454+0000","last_clean":"2024-09-13T11:32:55.257840+0000","last_became_active":"2024-09-13T11:33:16.277928+0000","last_became_peered":"2024-09-13T11:33:16.277928+0000","last_unstale":"2024-09-13T11:33:28.411454+0000","last_undegraded":"2024-09-13T11:33:16.272371+0000","last_fullsized":"2024-09-13T11:33:16.272253+0000","mapping_epoch":72,"log_start":"76'315","ondisk_log_start":"76'315","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":330,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:35.602877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14055929,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":290,"num_read_kb":5121,"num_write":410,"num_write_kb":81806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":16251716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3],"acting":[15,3],"avail_no_missing":["15","3"],"object_location_counts":[{"shards":"3,15","objects":35}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":106,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T11:33:28.411583+0000","last_change":"2024-09-13T11:33:27.080910+0000","last_active":"2024-09-13T11:33:28.411583+0000","last_peered":"2024-09-13T11:33:28.411583+0000","last_clean":"2024-09-13T11:33:28.411583+0000","last_became_active":"2024-09-13T11:32:45.284150+0000","last_became_peered":"2024-09-13T11:32:45.284150+0000","last_unstale":"2024-09-13T11:33:28.411583+0000","last_undegraded":"2024-09-13T11:33:28.411583+0000","last_fullsized":"2024-09-13T11:33:28.411583+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:33:27.080847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:27.080847+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:31.383253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,0],"acting":[15,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":237,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.642213+0000","last_change":"2024-09-13T11:33:16.944043+0000","last_active":"2024-09-13T11:33:26.642213+0000","last_peered":"2024-09-13T11:33:26.642213+0000","last_clean":"2024-09-13T11:33:26.642213+0000","last_became_active":"2024-09-13T11:32:48.286550+0000","last_became_peered":"2024-09-13T11:32:48.286550+0000","last_unstale":"2024-09-13T11:33:26.642213+0000","last_undegraded":"2024-09-13T11:33:26.642213+0000","last_fullsized":"2024-09-13T11:33:26.642213+0000","mapping_epoch":52,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:33:16.943930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:33:16.943930+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:23.725756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"76'46","reported_seq":237,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.485118+0000","last_change":"2024-09-13T11:33:02.151042+0000","last_active":"2024-09-13T11:33:26.485118+0000","last_peered":"2024-09-13T11:33:26.485118+0000","last_clean":"2024-09-13T11:33:26.485118+0000","last_became_active":"2024-09-13T11:32:54.609316+0000","last_became_peered":"2024-09-13T11:32:54.609316+0000","last_unstale":"2024-09-13T11:33:26.485118+0000","last_undegraded":"2024-09-13T11:33:26.485118+0000","last_fullsized":"2024-09-13T11:33:26.485118+0000","mapping_epoch":50,"log_start":"76'43","ondisk_log_start":"76'43","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'41","last_scrub_stamp":"2024-09-13T11:33:02.150975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.150975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":42,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:24.325318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311012,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"61'53","reported_seq":252,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.315026+0000","last_change":"2024-09-13T11:33:01.646865+0000","last_active":"2024-09-13T11:33:26.315026+0000","last_peered":"2024-09-13T11:33:26.315026+0000","last_clean":"2024-09-13T11:33:26.315026+0000","last_became_active":"2024-09-13T11:32:54.601608+0000","last_became_peered":"2024-09-13T11:32:54.601608+0000","last_unstale":"2024-09-13T11:33:26.315026+0000","last_undegraded":"2024-09-13T11:33:26.315026+0000","last_fullsized":"2024-09-13T11:33:26.315026+0000","mapping_epoch":31,"log_start":"61'50","ondisk_log_start":"61'50","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'53","last_scrub_stamp":"2024-09-13T11:33:01.646779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.646779+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:10.069513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546277,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":57613,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"76'49","reported_seq":240,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.642054+0000","last_change":"2024-09-13T11:33:02.042709+0000","last_active":"2024-09-13T11:33:26.642054+0000","last_peered":"2024-09-13T11:33:26.642054+0000","last_clean":"2024-09-13T11:33:26.642054+0000","last_became_active":"2024-09-13T11:32:54.604804+0000","last_became_peered":"2024-09-13T11:32:54.604804+0000","last_unstale":"2024-09-13T11:33:26.642054+0000","last_undegraded":"2024-09-13T11:33:26.642054+0000","last_fullsized":"2024-09-13T11:33:26.642054+0000","mapping_epoch":31,"log_start":"76'46","ondisk_log_start":"76'46","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2024-09-13T11:33:02.042611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.042611+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":46,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:02.206008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764867,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":59787,"num_write":18,"num_write_kb":5439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14,7],"acting":[0,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"77'70","reported_seq":259,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.485237+0000","last_change":"2024-09-13T11:32:47.284684+0000","last_active":"2024-09-13T11:33:26.485237+0000","last_peered":"2024-09-13T11:33:26.485237+0000","last_clean":"2024-09-13T11:33:26.485237+0000","last_became_active":"2024-09-13T11:32:47.279408+0000","last_became_peered":"2024-09-13T11:32:47.279408+0000","last_unstale":"2024-09-13T11:33:26.485237+0000","last_undegraded":"2024-09-13T11:33:26.485237+0000","last_fullsized":"2024-09-13T11:33:26.485237+0000","mapping_epoch":51,"log_start":"75'67","ondisk_log_start":"75'67","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":67,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.508020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941258,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":115824,"num_write":77,"num_write_kb":20518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2979858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"76'68","reported_seq":260,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.485192+0000","last_change":"2024-09-13T11:32:48.288341+0000","last_active":"2024-09-13T11:33:26.485192+0000","last_peered":"2024-09-13T11:33:26.485192+0000","last_clean":"2024-09-13T11:33:26.485192+0000","last_became_active":"2024-09-13T11:32:48.283697+0000","last_became_peered":"2024-09-13T11:32:48.283697+0000","last_unstale":"2024-09-13T11:33:26.485192+0000","last_undegraded":"2024-09-13T11:33:26.485192+0000","last_fullsized":"2024-09-13T11:33:26.485192+0000","mapping_epoch":52,"log_start":"76'65","ondisk_log_start":"76'65","created":31,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":65,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:46.169371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302691,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":167,"num_read_kb":147744,"num_write":70,"num_write_kb":20470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4770005,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"78'37","reported_seq":211,"reported_epoch":79,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:28.411592+0000","last_change":"2024-09-13T11:33:27.445927+0000","last_active":"2024-09-13T11:33:28.411592+0000","last_peered":"2024-09-13T11:33:28.411592+0000","last_clean":"2024-09-13T11:32:55.276844+0000","last_became_active":"2024-09-13T11:32:57.282855+0000","last_became_peered":"2024-09-13T11:32:57.282855+0000","last_unstale":"2024-09-13T11:33:28.411592+0000","last_undegraded":"2024-09-13T11:33:27.445134+0000","last_fullsized":"2024-09-13T11:32:57.277023+0000","mapping_epoch":59,"log_start":"78'34","ondisk_log_start":"78'34","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":34,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:30.132875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":56905,"num_write":35,"num_write_kb":11475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2040825,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7],"acting":[15,7],"avail_no_missing":["15","7"],"object_location_counts":[{"shards":"7,15","objects":1}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.d","version":"76'93","reported_seq":307,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T11:33:26.315135+0000","last_change":"2024-09-13T11:33:16.280221+0000","last_active":"2024-09-13T11:33:26.315135+0000","last_peered":"2024-09-13T11:33:26.315135+0000","last_clean":"2024-09-13T11:33:26.315135+0000","last_became_active":"2024-09-13T11:33:16.279717+0000","last_became_peered":"2024-09-13T11:33:16.279717+0000","last_unstale":"2024-09-13T11:33:26.315135+0000","last_undegraded":"2024-09-13T11:33:26.315135+0000","last_fullsized":"2024-09-13T11:33:26.315135+0000","mapping_epoch":72,"log_start":"76'90","ondisk_log_start":"76'90","created":31,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":90,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:48.989249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9445600,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":248,"num_read_kb":183108,"num_write":95,"num_write_kb":26413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"77'85","reported_seq":361,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.102658+0000","last_change":"2024-09-13T11:32:47.281013+0000","last_active":"2024-09-13T11:33:29.102658+0000","last_peered":"2024-09-13T11:33:29.102658+0000","last_clean":"2024-09-13T11:33:29.102658+0000","last_became_active":"2024-09-13T11:32:47.277862+0000","last_became_peered":"2024-09-13T11:32:47.277862+0000","last_unstale":"2024-09-13T11:33:29.102658+0000","last_undegraded":"2024-09-13T11:33:29.102658+0000","last_fullsized":"2024-09-13T11:33:29.102658+0000","mapping_epoch":51,"log_start":"76'82","ondisk_log_start":"76'82","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":82,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.750825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8806934,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":337,"num_read_kb":264822,"num_write":86,"num_write_kb":26021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5522961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,0],"acting":[15,14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.f","version":"76'103","reported_seq":370,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:26.642461+0000","last_change":"2024-09-13T11:32:57.287440+0000","last_active":"2024-09-13T11:33:26.642461+0000","last_peered":"2024-09-13T11:33:26.642461+0000","last_clean":"2024-09-13T11:32:55.848229+0000","last_became_active":"2024-09-13T11:32:57.287440+0000","last_became_peered":"2024-09-13T11:32:57.287440+0000","last_unstale":"2024-09-13T11:33:26.642461+0000","last_undegraded":"2024-09-13T11:32:57.280313+0000","last_fullsized":"2024-09-13T11:32:57.280204+0000","mapping_epoch":59,"log_start":"75'99","ondisk_log_start":"75'99","created":31,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":99,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:33.810129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7647284,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":314,"num_read_kb":316581,"num_write":105,"num_write_kb":28552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5762564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":["10","14"],"object_location_counts":[{"shards":"10,14","objects":3}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":116759367,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":2474,"num_read_kb":2129661,"num_write":892,"num_write_kb":263857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":27960674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":302403584,"data_stored":301909605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":64,"acting":72,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":87320370,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":1012,"num_read_kb":27251,"num_write":1447,"num_write_kb":303757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":83728348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":343449600,"data_stored":342581069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":9,"acting":10,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":22,"acting":22,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112948,"kb_used_data":81764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94258892,"statfs":{"total":96636764160,"available":96521105408,"internally_reserved":0,"allocated":83726336,"data_stored":83116628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":98784247828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247828,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83692,"kb_used_data":52508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288148,"statfs":{"total":96636764160,"available":96551063552,"internally_reserved":0,"allocated":53768192,"data_stored":53279411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280531,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79944,"kb_used_data":48760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94291896,"statfs":{"total":96636764160,"available":96554901504,"internally_reserved":0,"allocated":49930240,"data_stored":49457941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280531,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280531,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":24068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":24645632,"data_stored":24246714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":170144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93172,"kb_used_data":61988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94278668,"statfs":{"total":96636764160,"available":96541356032,"internally_reserved":0,"allocated":63475712,"data_stored":62982760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,4,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280531,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48152,"kb_used_data":21128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323688,"statfs":{"total":96636764160,"available":96587456512,"internally_reserved":0,"allocated":21635072,"data_stored":21238948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":94489280531,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280532,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95272,"kb_used_data":64088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94276568,"statfs":{"total":96636764160,"available":96539205632,"internally_reserved":0,"allocated":65626112,"data_stored":65107154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43128,"kb_used_data":16104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328712,"statfs":{"total":96636764160,"available":96592601088,"internally_reserved":0,"allocated":16490496,"data_stored":16112354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280531,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280532,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45740,"kb_used_data":18716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94326100,"statfs":{"total":96636764160,"available":96589926400,"internally_reserved":0,"allocated":19165184,"data_stored":18770851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313235,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63368,"kb_used_data":36344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94308472,"statfs":{"total":96636764160,"available":96571875328,"internally_reserved":0,"allocated":37216256,"data_stored":36754096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,6,7,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95324,"kb_used_data":68300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276516,"statfs":{"total":96636764160,"available":96539152384,"internally_reserved":0,"allocated":69939200,"data_stored":69555056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23789568,"data_stored":23723612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24444928,"data_stored":24413026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23801856,"data_stored":23781575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14110720,"data_stored":14055929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":52498432,"data_stored":52355762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":23068672,"data_stored":22992714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28405760,"data_stored":28348291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":23789568,"data_stored":23723612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":42668032,"data_stored":42597076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":23068672,"data_stored":22992714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":63803392,"data_stored":63596758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24674304,"data_stored":24631308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18067456,"data_stored":18030231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4225584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46501888,"data_stored":46432687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20762624,"data_stored":20726295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2412544,"data_stored":2410759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12255232,"data_stored":12238739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":29802496,"data_stored":29757228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15036416,"data_stored":15017741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":3870720,"data_stored":3863931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":18288640,"data_stored":18258179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12550144,"data_stored":12517812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":26669056,"data_stored":26630598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":24473600,"data_stored":24444051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":23773184,"data_stored":23734061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":19025920,"data_stored":18990401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:33:31.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 4.16 3.0 4.6 4.c 4.f 2024-09-13T11:33:31.338 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:31.337+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:31.345 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2503911 2024-09-13T11:33:31.354 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:31.355 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:31.355 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:31.355 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:31.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:33:31.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:31.364 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:31.365 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:31.384 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:31.384 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:31.385 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:31.404 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:31.404 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:31.420 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:31.421 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:31.438 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:31.438 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:31.439 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:31.438+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:31.448 INFO:tasks.rados.rados.0.smithi204.stdout:1646: expect (ObjNum 330 snap 0 seq_num 330) 2024-09-13T11:33:31.448 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:31.457 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:31.458 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:31.471 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:31.471 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:31.474 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:31.475 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:31.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:31.501 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:31.502 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:31.518 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:31.518 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:31.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:31.536 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:31.540 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:31.538+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:31.552 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:31.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:31.553 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:31.567 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2720685 2024-09-13T11:33:31.569 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:31.570 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:31.585 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:31.585 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:31.586 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:31.586 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:31.589 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:31.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:31.604 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:31.605 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:31.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:31.638 INFO:teuthology.orchestra.run.smithi038.stderr:instructing pg(s) [4.6,4.f,4.16] on osd.10 to force-backfill; instructing pg(s) [3.0,4.c] on osd.15 to force-backfill; 2024-09-13T11:33:31.638 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:33:31.640 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:31.640+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:31.668 INFO:tasks.rados.rados.0.smithi204.stdout:1652: finishing write tid 1 to smithi20438004-7 2024-09-13T11:33:31.668 INFO:tasks.rados.rados.0.smithi204.stdout:1652: finishing write tid 2 to smithi20438004-7 2024-09-13T11:33:31.668 INFO:tasks.rados.rados.0.smithi204.stdout:1644: done (9 left) 2024-09-13T11:33:31.668 INFO:tasks.rados.rados.0.smithi204.stdout:1645: done (8 left) 2024-09-13T11:33:31.668 INFO:tasks.rados.rados.0.smithi204.stdout:1646: done (7 left) 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1647: done (6 left) 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1658: write oid 14 current snap is 0 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1652: finishing write tid 3 to smithi20438004-7 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1652: finishing write tid 4 to smithi20438004-7 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1652: finishing write tid 5 to smithi20438004-7 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1652: finishing write tid 6 to smithi20438004-7 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1652: oid 7 updating version 0 to 82 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1652: oid 7 version 82 is already newer than 80 2024-09-13T11:33:31.669 INFO:tasks.rados.rados.0.smithi204.stdout:1652: oid 7 version 82 is already newer than 81 2024-09-13T11:33:31.670 INFO:tasks.rados.rados.0.smithi204.stdout:1652: oid 7 updating version 82 to 83 2024-09-13T11:33:31.670 INFO:tasks.rados.rados.0.smithi204.stdout:1652: oid 7 version 83 is already newer than 79 2024-09-13T11:33:31.670 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 83 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-13T11:33:31.670 INFO:tasks.rados.rados.0.smithi204.stdout:1652: left oid 7 (ObjNum 342 snap 0 seq_num 342) 2024-09-13T11:33:31.670 INFO:tasks.rados.rados.0.smithi204.stdout:1653: expect (ObjNum 260 snap 0 seq_num 260) 2024-09-13T11:33:31.670 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:31.738 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:31.738 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:31.741 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:31.740+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:31.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:31.803 INFO:tasks.rados.rados.0.smithi204.stdout:1654: finishing write tid 1 to smithi20438004-5 2024-09-13T11:33:31.804 INFO:tasks.rados.rados.0.smithi204.stdout:1654: finishing write tid 2 to smithi20438004-5 2024-09-13T11:33:31.804 INFO:tasks.rados.rados.0.smithi204.stdout:1654: finishing write tid 3 to smithi20438004-5 2024-09-13T11:33:31.804 INFO:tasks.rados.rados.0.smithi204.stdout:1658: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-13T11:33:31.810 INFO:tasks.rados.rados.0.smithi204.stdout:1658: writing smithi20438004-14 from 581608 to 1303149 tid 1 2024-09-13T11:33:31.816 INFO:tasks.rados.rados.0.smithi204.stdout:1658: writing smithi20438004-14 from 1863688 to 2349461 tid 2 2024-09-13T11:33:31.824 INFO:tasks.rados.rados.0.smithi204.stdout:1658: writing smithi20438004-14 from 3113980 to 3718023 tid 3 2024-09-13T11:33:31.824 INFO:tasks.rados.rados.0.smithi204.stdout:1652: done (6 left) 2024-09-13T11:33:31.824 INFO:tasks.rados.rados.0.smithi204.stdout:1653: done (5 left) 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1659: read oid 22 snap -1 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1659: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1660: read oid 13 snap -1 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1660: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1661: read oid 43 snap -1 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1661: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1662: write (excl) oid 46 current snap is 0 2024-09-13T11:33:31.825 INFO:tasks.rados.rados.0.smithi204.stdout:1662: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-13T11:33:31.832 INFO:tasks.rados.rados.0.smithi204.stdout:1662: writing smithi20438004-46 from 598415 to 1395205 tid 1 2024-09-13T11:33:31.839 INFO:tasks.rados.rados.0.smithi204.stdout:1662: writing smithi20438004-46 from 1922170 to 2667954 tid 2 2024-09-13T11:33:31.842 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:31.841+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:31.845 INFO:tasks.rados.rados.0.smithi204.stdout:1662: writing smithi20438004-46 from 3141403 to 3556718 tid 3 2024-09-13T11:33:31.845 INFO:tasks.rados.rados.0.smithi204.stdout:1663: tier_promote oid 1 2024-09-13T11:33:31.845 INFO:tasks.rados.rados.0.smithi204.stdout:1664: set_chunk oid 26 target oid 2024-09-13T11:33:31.846 INFO:tasks.rados.rados.0.smithi204.stdout:1664: set_chunk oid 26 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:31.846 INFO:tasks.rados.rados.0.smithi204.stdout:1665: read oid 20 snap -1 2024-09-13T11:33:31.846 INFO:tasks.rados.rados.0.smithi204.stdout:1665: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:31.846 INFO:tasks.rados.rados.0.smithi204.stdout:1666: write oid 25 current snap is 0 2024-09-13T11:33:31.846 INFO:tasks.rados.rados.0.smithi204.stdout:1666: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-13T11:33:31.852 INFO:tasks.rados.rados.0.smithi204.stdout:1666: writing smithi20438004-25 from 615222 to 1087261 tid 1 2024-09-13T11:33:31.857 INFO:tasks.rados.rados.0.smithi204.stdout:1666: writing smithi20438004-25 from 1580652 to 2070094 tid 2 2024-09-13T11:33:31.865 INFO:tasks.rados.rados.0.smithi204.stdout:1666: writing smithi20438004-25 from 2768826 to 3395413 tid 3 2024-09-13T11:33:31.865 INFO:tasks.rados.rados.0.smithi204.stdout:1667: read oid 50 snap -1 2024-09-13T11:33:31.865 INFO:tasks.rados.rados.0.smithi204.stdout:1667: expect deleted 2024-09-13T11:33:31.865 INFO:tasks.rados.rados.0.smithi204.stdout:1668: read oid 40 snap -1 2024-09-13T11:33:31.866 INFO:tasks.rados.rados.0.smithi204.stdout:1668: expect (ObjNum 320 snap 0 seq_num 320) 2024-09-13T11:33:31.866 INFO:tasks.rados.rados.0.smithi204.stdout:1669: read oid 4 snap -1 2024-09-13T11:33:31.866 INFO:tasks.rados.rados.0.smithi204.stdout:1669: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-13T11:33:31.866 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:31.866 INFO:tasks.rados.rados.0.smithi204.stdout:1656: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-13T11:33:31.866 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 442997 2024-09-13T11:33:31.899 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:31.900 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:31.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:31.942 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:31.941+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:31.951 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:31.996 INFO:tasks.rados.rados.0.smithi204.stdout:1654: finishing write tid 4 to smithi20438004-5 2024-09-13T11:33:31.996 INFO:tasks.rados.rados.0.smithi204.stdout:1654: finishing write tid 5 to smithi20438004-5 2024-09-13T11:33:31.996 INFO:tasks.rados.rados.0.smithi204.stdout:1654: oid 5 updating version 0 to 81 2024-09-13T11:33:31.996 INFO:tasks.rados.rados.0.smithi204.stdout:1654: oid 5 updating version 81 to 82 2024-09-13T11:33:31.996 INFO:tasks.rados.rados.0.smithi204.stdout:1654: oid 5 version 82 is already newer than 80 2024-09-13T11:33:31.996 INFO:tasks.rados.rados.0.smithi204.stdout:1654: oid 5 updating version 82 to 83 2024-09-13T11:33:31.997 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 83 (ObjNum 343 snap 0 seq_num 343) dirty exists 2024-09-13T11:33:31.997 INFO:tasks.rados.rados.0.smithi204.stdout:1654: left oid 5 (ObjNum 343 snap 0 seq_num 343) 2024-09-13T11:33:31.997 INFO:tasks.rados.rados.0.smithi204.stdout:1657: got (0) Success 2024-09-13T11:33:31.997 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 119 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-13T11:33:31.997 INFO:tasks.rados.rados.0.smithi204.stdout:1655: expect (ObjNum 337 snap 0 seq_num 337) 2024-09-13T11:33:31.997 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:32.012 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.012 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:32.043 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:32.042+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:32.124 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.125 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:32.137 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:32.144 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:32.143+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:32.208 INFO:tasks.rados.rados.0.smithi204.stdout:1658: finishing write tid 1 to smithi20438004-14 2024-09-13T11:33:32.208 INFO:tasks.rados.rados.0.smithi204.stdout:1658: finishing write tid 2 to smithi20438004-14 2024-09-13T11:33:32.208 INFO:tasks.rados.rados.0.smithi204.stdout:1661: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:32.236 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.236 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:32.246 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:32.245+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:32.347 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:32.345+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:32.348 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.349 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.357 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:32.402 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:33:32.423 INFO:tasks.rados.rados.0.smithi204.stdout:1658: finishing write tid 3 to smithi20438004-14 2024-09-13T11:33:32.423 INFO:tasks.rados.rados.0.smithi204.stdout:1658: finishing write tid 4 to smithi20438004-14 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:1658: finishing write tid 5 to smithi20438004-14 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:1658: oid 14 updating version 0 to 123 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:1658: oid 14 updating version 123 to 126 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:1658: oid 14 version 126 is already newer than 124 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:1658: oid 14 version 126 is already newer than 125 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 126 (ObjNum 344 snap 0 seq_num 344) dirty exists 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:1658: left oid 14 (ObjNum 344 snap 0 seq_num 344) 2024-09-13T11:33:32.424 INFO:tasks.rados.rados.0.smithi204.stdout:1659: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-13T11:33:32.447 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:32.446+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:32.461 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.462 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.470 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:32.492 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 110833 2024-09-13T11:33:32.549 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:32.548+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:32.563 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 978393 2024-09-13T11:33:32.597 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.597 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.606 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:32.620 INFO:tasks.rados.rados.0.smithi204.stdout:1662: finishing write tid 1 to smithi20438004-46 2024-09-13T11:33:32.620 INFO:tasks.rados.rados.0.smithi204.stdout:1654: done (15 left) 2024-09-13T11:33:32.620 INFO:tasks.rados.rados.0.smithi204.stdout:1655: done (14 left) 2024-09-13T11:33:32.621 INFO:tasks.rados.rados.0.smithi204.stdout:1656: done (13 left) 2024-09-13T11:33:32.621 INFO:tasks.rados.rados.0.smithi204.stdout:1657: done (12 left) 2024-09-13T11:33:32.621 INFO:tasks.rados.rados.0.smithi204.stdout:1658: done (11 left) 2024-09-13T11:33:32.621 INFO:tasks.rados.rados.0.smithi204.stdout:1659: done (10 left) 2024-09-13T11:33:32.621 INFO:tasks.rados.rados.0.smithi204.stdout:1661: done (9 left) 2024-09-13T11:33:32.621 INFO:tasks.rados.rados.0.smithi204.stdout:1670: set_chunk oid 17 target oid 2024-09-13T11:33:32.630 INFO:tasks.rados.rados.0.smithi204.stdout:1670: set_chunk oid 17 offset: 1291264 length: 1230848 target oid acc8201d1888bd8d2d369ca7aca03fb2e4f86e3c166de660a7b2ba85c3e46f58 offset: 0 snap -1 2024-09-13T11:33:32.630 INFO:tasks.rados.rados.0.smithi204.stdout:1671: write (excl) oid 6 current snap is 0 2024-09-13T11:33:32.630 INFO:tasks.rados.rados.0.smithi204.stdout:1671: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-13T11:33:32.636 INFO:tasks.rados.rados.0.smithi204.stdout:1671: writing smithi20438004-6 from 632029 to 1179317 tid 1 2024-09-13T11:33:32.643 INFO:tasks.rados.rados.0.smithi204.stdout:1671: writing smithi20438004-6 from 1639134 to 2388587 tid 2 2024-09-13T11:33:32.648 INFO:tasks.rados.rados.0.smithi204.stdout:1671: writing smithi20438004-6 from 2796249 to 3234108 tid 3 2024-09-13T11:33:32.650 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:32.648+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:32.651 INFO:tasks.rados.rados.0.smithi204.stdout:1671: writing smithi20438004-6 from 3653524 to 3832029 tid 4 2024-09-13T11:33:32.651 INFO:tasks.rados.rados.0.smithi204.stdout:1672: set_chunk oid 45 target oid 2024-09-13T11:33:32.662 INFO:tasks.rados.rados.0.smithi204.stdout:1672: set_chunk oid 45 offset: 1605632 length: 1530368 target oid 4740467b29716747f9bc558592fd595e40baea9fccffee210178d50681dcc606 offset: 0 snap -1 2024-09-13T11:33:32.663 INFO:tasks.rados.rados.0.smithi204.stdout:1673: read oid 10 snap -1 2024-09-13T11:33:32.663 INFO:tasks.rados.rados.0.smithi204.stdout:1673: expect (ObjNum 271 snap 0 seq_num 271) 2024-09-13T11:33:32.663 INFO:tasks.rados.rados.0.smithi204.stdout:1674: read oid 14 snap -1 2024-09-13T11:33:32.663 INFO:tasks.rados.rados.0.smithi204.stdout:1674: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-13T11:33:32.663 INFO:tasks.rados.rados.0.smithi204.stdout:1675: write (excl) oid 35 current snap is 0 2024-09-13T11:33:32.663 INFO:tasks.rados.rados.0.smithi204.stdout:1675: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-13T11:33:32.669 INFO:tasks.rados.rados.0.smithi204.stdout:1675: writing smithi20438004-35 from 648836 to 1271373 tid 1 2024-09-13T11:33:32.676 INFO:tasks.rados.rados.0.smithi204.stdout:1675: writing smithi20438004-35 from 1981263 to 2474374 tid 2 2024-09-13T11:33:32.684 INFO:tasks.rados.rados.0.smithi204.stdout:1675: writing smithi20438004-35 from 3107319 to 3848836 tid 3 2024-09-13T11:33:32.684 INFO:tasks.rados.rados.0.smithi204.stdout:1676: read oid 34 snap -1 2024-09-13T11:33:32.684 INFO:tasks.rados.rados.0.smithi204.stdout:1676: expect (ObjNum 340 snap 0 seq_num 340) 2024-09-13T11:33:32.685 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:32.685 INFO:tasks.rados.rados.0.smithi204.stdout:1660: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:32.722 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.722 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.731 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:32.751 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:32.750+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:32.766 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 13608 2024-09-13T11:33:32.852 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:32.851+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:32.870 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:32.870 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:32.879 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:32.922 INFO:tasks.rados.rados.0.smithi204.stdout:1662: finishing write tid 2 to smithi20438004-46 2024-09-13T11:33:32.922 INFO:tasks.rados.rados.0.smithi204.stdout:1664: finishing set_chunk to oid 26 2024-09-13T11:33:32.922 INFO:tasks.rados.rados.0.smithi204.stdout:1664: got expected ENOENT (src dne) 2024-09-13T11:33:32.922 INFO:tasks.rados.rados.0.smithi204.stdout:1662: finishing write tid 3 to smithi20438004-46 2024-09-13T11:33:32.922 INFO:tasks.rados.rados.0.smithi204.stdout:1660: done (15 left) 2024-09-13T11:33:32.922 INFO:tasks.rados.rados.0.smithi204.stdout:1664: done (14 left) 2024-09-13T11:33:32.922 INFO:tasks.rados.rados.0.smithi204.stdout:1677: read oid 8 snap -1 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1677: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1678: read oid 23 snap -1 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1678: expect deleted 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1662: finishing write tid 4 to smithi20438004-46 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1662: finishing write tid 5 to smithi20438004-46 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1662: oid 46 updating version 0 to 86 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1662: oid 46 updating version 86 to 88 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1662: oid 46 version 88 is already newer than 87 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1662: oid 46 updating version 88 to 89 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 89 (ObjNum 345 snap 0 seq_num 345) dirty exists 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1662: left oid 46 (ObjNum 345 snap 0 seq_num 345) 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:1665: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-13T11:33:32.923 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 91852 2024-09-13T11:33:32.955 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:32.953+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:33.025 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:33.025 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:33.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:33.034 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2816585 2024-09-13T11:33:33.043 INFO:tasks.rados.rados.0.smithi204.stdout:1666: finishing write tid 1 to smithi20438004-25 2024-09-13T11:33:33.043 INFO:tasks.rados.rados.0.smithi204.stdout:1663: got (0) Success 2024-09-13T11:33:33.043 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 78 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-09-13T11:33:33.043 INFO:tasks.rados.rados.0.smithi204.stdout:1666: finishing write tid 2 to smithi20438004-25 2024-09-13T11:33:33.043 INFO:tasks.rados.rados.0.smithi204.stdout:1666: finishing write tid 3 to smithi20438004-25 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1666: finishing write tid 4 to smithi20438004-25 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1666: finishing write tid 5 to smithi20438004-25 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1666: oid 25 updating version 0 to 114 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1666: oid 25 updating version 114 to 117 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1666: oid 25 version 117 is already newer than 115 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1666: oid 25 version 117 is already newer than 116 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 117 (ObjNum 346 snap 0 seq_num 346) dirty exists 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1666: left oid 25 (ObjNum 346 snap 0 seq_num 346) 2024-09-13T11:33:33.044 INFO:tasks.rados.rados.0.smithi204.stdout:1668: expect (ObjNum 320 snap 0 seq_num 320) 2024-09-13T11:33:33.054 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:33.053+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:33.087 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3305904 2024-09-13T11:33:33.137 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:33.137 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:33.146 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:33.155 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:33.154+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:33.231 INFO:tasks.rados.rados.0.smithi204.stdout:1669: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-13T11:33:33.232 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:33.256 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:33.255+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:33.286 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:33.291 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:33.291 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:33.300 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:33.341 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 111405 2024-09-13T11:33:33.357 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:33.356+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:33.393 INFO:tasks.rados.rados.0.smithi204.stdout:1670: finishing set_chunk to oid 17 2024-09-13T11:33:33.393 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 17 set_chunk returned error code -95 offset: 1291264 length: 1230848 2024-09-13T11:33:33.393 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 0 (ObjNum 289 snap 0 seq_num 289) dirty exists 2024-09-13T11:33:33.393 INFO:tasks.rados.rados.0.smithi204.stdout:1671: finishing write tid 1 to smithi20438004-6 2024-09-13T11:33:33.393 INFO:tasks.rados.rados.0.smithi204.stdout:1671: finishing write tid 2 to smithi20438004-6 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1662: done (15 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1663: done (14 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1665: done (13 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1666: done (12 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1667: done (11 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1668: done (10 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1669: done (9 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1670: done (8 left) 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1679: write (excl) oid 37 current snap is 0 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1671: finishing write tid 3 to smithi20438004-6 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1671: finishing write tid 4 to smithi20438004-6 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1671: finishing write tid 5 to smithi20438004-6 2024-09-13T11:33:33.394 INFO:tasks.rados.rados.0.smithi204.stdout:1679: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-09-13T11:33:33.400 INFO:tasks.rados.rados.0.smithi204.stdout:1679: writing smithi20438004-37 from 665643 to 1363429 tid 1 2024-09-13T11:33:33.403 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:33.403 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:33.408 INFO:tasks.rados.rados.0.smithi204.stdout:1679: writing smithi20438004-37 from 2039745 to 2792867 tid 2 2024-09-13T11:33:33.411 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:33.415 INFO:tasks.rados.rados.0.smithi204.stdout:1679: writing smithi20438004-37 from 3534742 to 3865643 tid 3 2024-09-13T11:33:33.415 INFO:tasks.rados.rados.0.smithi204.stdout:1680: read oid 25 snap -1 2024-09-13T11:33:33.415 INFO:tasks.rados.rados.0.smithi204.stdout:1680: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-13T11:33:33.415 INFO:tasks.rados.rados.0.smithi204.stdout:1681: read oid 19 snap -1 2024-09-13T11:33:33.415 INFO:tasks.rados.rados.0.smithi204.stdout:1681: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:33.415 INFO:tasks.rados.rados.0.smithi204.stdout:1682: read oid 5 snap -1 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1682: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1683: write (excl) oid 50 current snap is 0 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1671: finishing write tid 6 to smithi20438004-6 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1671: oid 6 updating version 0 to 95 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1671: oid 6 version 95 is already newer than 93 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1671: oid 6 version 95 is already newer than 92 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1671: oid 6 updating version 95 to 96 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1671: oid 6 version 96 is already newer than 94 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 96 (ObjNum 347 snap 0 seq_num 347) dirty exists 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1671: left oid 6 (ObjNum 347 snap 0 seq_num 347) 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1672: finishing set_chunk to oid 45 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 78 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-09-13T11:33:33.416 INFO:tasks.rados.rados.0.smithi204.stdout:1673: expect (ObjNum 271 snap 0 seq_num 271) 2024-09-13T11:33:33.426 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2171970 2024-09-13T11:33:33.431 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 87100 2024-09-13T11:33:33.458 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:33.458+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:33.490 INFO:tasks.rados.rados.0.smithi204.stdout:1674: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-13T11:33:33.521 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:33.521 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:33.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:33.571 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:33.570+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:33.581 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:33.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:33.660 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:33.659+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:33.674 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:33.674 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:33.683 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: finishing write tid 1 to smithi20438004-35 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: finishing write tid 2 to smithi20438004-35 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: finishing write tid 3 to smithi20438004-35 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: finishing write tid 4 to smithi20438004-35 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: finishing write tid 5 to smithi20438004-35 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: oid 35 updating version 0 to 140 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: oid 35 version 140 is already newer than 137 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: oid 35 version 140 is already newer than 139 2024-09-13T11:33:33.736 INFO:tasks.rados.rados.0.smithi204.stdout:1675: oid 35 version 140 is already newer than 138 2024-09-13T11:33:33.737 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 140 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-09-13T11:33:33.737 INFO:tasks.rados.rados.0.smithi204.stdout:1675: left oid 35 (ObjNum 348 snap 0 seq_num 348) 2024-09-13T11:33:33.737 INFO:tasks.rados.rados.0.smithi204.stdout:1676: expect (ObjNum 340 snap 0 seq_num 340) 2024-09-13T11:33:33.748 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:33.749 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:33.761 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:33.760+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:33.767 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:33.767 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:33.767 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:33.784 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:33.785 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:33.800 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:33.801 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:33.813 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:33.818 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:33.818 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:33.827 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:33.827 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:33.835 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:33.835 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:33.836 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:33.853 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:33.853 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:33.862 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:33.860+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:33.869 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:33.869 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:33.886 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:33.886 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:33.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:33.920 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:33.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:33.921 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:33.937 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:33.938 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:33.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:33.955 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:33.962 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:33.961+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:33.964 INFO:tasks.rados.rados.0.smithi204.stdout:1677: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:33.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:33:33.970+0000 7f4fe5191640 0 -- 172.21.15.38:0/237535660 >> v2:172.21.15.204:6812/501019151 conn(0x7f4fe021f620 msgr2=0x7f4fe0221a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:33:33.973 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:33.977 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:33.982 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.888952356000004, 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:33.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:33.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:33:33.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:33:33.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:33:33.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:33:33.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.878677788999994, 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:33:33.987 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:33:33.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:33:33.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.866009887999994, 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:33:33.995 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.859400041000001, 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:33.996 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:33:33.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:33.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:33.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:34.000 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.851023010999995, 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:34.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:33:34.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:33:34.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.825891831999996, 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:33:34.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:33:34.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.816990554, 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.009 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:33:34.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:33:34.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:33:34.013 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.809392830999997, 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.017 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.803988902, 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:33:34.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.791365212000002, 2024-09-13T11:33:34.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:33:34.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:33:34.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.777185762000002, 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:33:34.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.768967594000003, 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:33:34.031 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:33:34.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:33:34.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:33:34.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.738144191000004, 2024-09-13T11:33:34.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:33:34.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:33:34.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.043 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 66.591429403999996, 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:33:34.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:33:34.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:33:34.047 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 57.886257352000001, 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.048 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.049 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:33:34.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 57.138304982000001, 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013151014000000001, 2024-09-13T11:33:34.052 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842045+0000", 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865e-05 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848182+0000", 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061372370000000002 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848205+0000", 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.298e-05 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848218+0000", 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.348e-05 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848300+0000", 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1766000000000003e-05 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848327+0000", 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6446999999999999e-05 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855119+0000", 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067926490000000004 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855177+0000", 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7590000000000003e-05 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.056 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1458 4.e 4:7984cf84:::smithi20438004-46:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 56.952692929999998, 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012793834, 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.057 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1458 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027640+0000", 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.046e-06 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027651+0000", 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.102e-05 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027669+0000", 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:33:34.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027681+0000", 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2177999999999999e-05 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027762+0000", 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0331e-05 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027868+0000", 2024-09-13T11:33:34.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106443 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.040432+0000", 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012563632999999999 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1787 4.e 4:7022fc62:::smithi20438004-6:head [read 188928~1265152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:34.061 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 46.706600238999997, 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013752110999999999, 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1787 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:34.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273731+0000", 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273734+0000", 2024-09-13T11:33:34.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3510000000000002e-06 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273741+0000", 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273787+0000", 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6198000000000001e-05 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274432+0000", 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064478300000000001 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274461+0000", 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9051999999999999e-05 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281751+0000", 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007289963 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281984+0000", 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023257099999999999 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.287483+0000", 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054989829999999998 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.066 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2762 4.e 4:7984cf84:::smithi20438004-46:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e66)", 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 27.084142821, 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022335361000000001, 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2762 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.067 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896189+0000", 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3519999999999999e-06 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.068 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896196+0000", 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896210+0000", 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3439e-05 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896217+0000", 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5889999999999997e-06 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.069 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896278+0000", 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1072000000000001e-05 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896338+0000", 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9998999999999999e-05 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.918523+0000", 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022185099999999999 2024-09-13T11:33:34.070 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3763 4.1 4:8533c921:::smithi20438004-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.8465878199999999, 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010966191, 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:34.071 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3763 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.072 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133745+0000", 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.75e-06 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133752+0000", 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133767+0000", 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4577e-05 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.073 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133774+0000", 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133836+0000", 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1787999999999999e-05 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133896+0000", 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9768000000000001e-05 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:34.074 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.144709+0000", 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010813285000000001 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:34.075 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:34.077 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2420654 2024-09-13T11:33:34.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:34.077 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:34.076+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:34.119 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:34.178 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:34.176+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:34.184 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:34.184 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:34.185 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:34.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:1679: finishing write tid 1 to smithi20438004-37 2024-09-13T11:33:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:1679: finishing write tid 2 to smithi20438004-37 2024-09-13T11:33:34.202 INFO:tasks.rados.rados.0.smithi204.stdout:1683: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-13T11:33:34.208 INFO:tasks.rados.rados.0.smithi204.stdout:1683: writing smithi20438004-50 from 682450 to 1171838 tid 1 2024-09-13T11:33:34.215 INFO:tasks.rados.rados.0.smithi204.stdout:1683: writing smithi20438004-50 from 1814580 to 2311360 tid 2 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1683: writing smithi20438004-50 from 2878518 to 3666546 tid 3 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1671: done (12 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1672: done (11 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1673: done (10 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1674: done (9 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1675: done (8 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1676: done (7 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1677: done (6 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1678: done (5 left) 2024-09-13T11:33:34.223 INFO:tasks.rados.rados.0.smithi204.stdout:1684: write (excl) oid 38 current snap is 0 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: finishing write tid 3 to smithi20438004-37 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: finishing write tid 4 to smithi20438004-37 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: finishing write tid 5 to smithi20438004-37 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: oid 37 updating version 0 to 42 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: oid 37 updating version 42 to 45 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: oid 37 version 45 is already newer than 44 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: oid 37 version 45 is already newer than 43 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 45 (ObjNum 349 snap 0 seq_num 349) dirty exists 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1679: left oid 37 (ObjNum 349 snap 0 seq_num 349) 2024-09-13T11:33:34.224 INFO:tasks.rados.rados.0.smithi204.stdout:1684: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-13T11:33:34.229 INFO:tasks.rados.rados.0.smithi204.stdout:1684: writing smithi20438004-38 from 699257 to 1263894 tid 1 2024-09-13T11:33:34.236 INFO:tasks.rados.rados.0.smithi204.stdout:1684: writing smithi20438004-38 from 1873062 to 2513500 tid 2 2024-09-13T11:33:34.244 INFO:tasks.rados.rados.0.smithi204.stdout:1684: writing smithi20438004-38 from 3305941 to 3899257 tid 3 2024-09-13T11:33:34.244 INFO:tasks.rados.rados.0.smithi204.stdout:1679: done (5 left) 2024-09-13T11:33:34.244 INFO:tasks.rados.rados.0.smithi204.stdout:1685: write oid 22 current snap is 0 2024-09-13T11:33:34.244 INFO:tasks.rados.rados.0.smithi204.stdout:1682: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-13T11:33:34.278 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:34.277+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:34.280 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:34.302 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:34.302 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:34.302 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:34.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:34.356 INFO:tasks.rados.rados.0.smithi204.stdout:1680: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-13T11:33:34.357 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:34.379 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:34.378+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:34.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:33:34.419 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 88.679938419999999, 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:33:34.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:33:34.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:139 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'105, mlcod=24'100)", 2024-09-13T11:33:34.424 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 88.679374816999996, 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061709499999999997, 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737743+0000", 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737744+0000", 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7700000000000004e-07 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737749+0000", 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5439999999999998e-06 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737756+0000", 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7649999999999996e-06 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737760+0000", 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6650000000000001e-06 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742613+0000", 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048528909999999998 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.742629+0000", 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6481e-05 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743902+0000", 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012723630000000001 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743914+0000", 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2663999999999999e-05 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.429 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.940012792999994, 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:33:34.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:34.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:33:34.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:33:34.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.934954235000006, 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:33:34.438 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:33:34.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.934530413000005, 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:33:34.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.934512369000004, 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.451 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:33:34.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:33:34.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.456 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.218271596999998, 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:33:34.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:33:34.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:33:34.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.464 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.204430807999998, 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:33:34.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:33:34.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.105462833999994, 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:33:34.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:33:34.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.478 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1751 4.9 4:9441befd:::smithi20438004-8:head [write 420208~530210,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.031550054, 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060468199999999996, 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.479 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1751 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385568+0000", 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.385569+0000", 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.426e-06 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386470+0000", 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090069800000000002 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386485+0000", 2024-09-13T11:33:34.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5225000000000001e-05 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386495+0000", 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5880000000000004e-06 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386574+0000", 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8911000000000004e-05 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386698+0000", 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012372800000000001 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.386913+0000", 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021553499999999999 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.388673+0000", 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017596210000000001 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.389582+0000", 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090943100000000004 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391260+0000", 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00167782 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391570+0000", 2024-09-13T11:33:34.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030968499999999997 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.391615+0000", 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5151999999999997e-05 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1752 4.9 4:9441befd:::smithi20438004-8:head [write 1527054~766050] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:34.485 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.023753896999999, 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062506259999999996, 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1752 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:34.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393365+0000", 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0720000000000001e-06 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394108+0000", 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074250399999999995 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394120+0000", 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1990999999999999e-05 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394125+0000", 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7270000000000004e-06 2024-09-13T11:33:34.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394182+0000", 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6181000000000003e-05 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394248+0000", 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6848999999999996e-05 2024-09-13T11:33:34.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394333+0000", 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4970999999999997e-05 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396066+0000", 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001732546 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397394+0000", 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013282680000000001 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399415+0000", 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002021063 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399585+0000", 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016999699999999999 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399615+0000", 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9456999999999999e-05 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1754 4.9 4:9441befd:::smithi20438004-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.022946333999997, 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057025110000000004, 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1754 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.493 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394172+0000", 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2300000000000001e-07 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394175+0000", 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5189999999999999e-06 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394181+0000", 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6089999999999997e-06 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394187+0000", 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7400000000000001e-06 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395928+0000", 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017413470000000001 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.395983+0000", 2024-09-13T11:33:34.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399771+0000", 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003787786 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399814+0000", 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2755e-05 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399874+0000", 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0350999999999999e-05 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1959 4.e e53/51 4:7984cf84:::smithi20438004-46:head v 53'41, mlcod=52'38)", 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 44.228701565000001, 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057232109999999997, 2024-09-13T11:33:34.498 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:34.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188417+0000", 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0599999999999999e-07 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188422+0000", 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9409999999999998e-06 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188434+0000", 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2002e-05 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188441+0000", 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0090000000000001e-06 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188712+0000", 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027109600000000002 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.188724+0000", 2024-09-13T11:33:34.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1790999999999999e-05 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194092+0000", 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053680450000000001 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:50.194140+0000", 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7521000000000001e-05 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2036 4.e e54/51 4:76a15ccf:::smithi20438004-1:head v 54'46, mlcod=54'44)", 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:34.504 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.839153060000001, 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056625620000000003, 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577965+0000", 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.577966+0000", 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4200000000000005e-07 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578870+0000", 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090414100000000002 2024-09-13T11:33:34.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578880+0000", 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0501e-05 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578886+0000", 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.356e-06 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578950+0000", 2024-09-13T11:33:34.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4045000000000001e-05 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.578968+0000", 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7935999999999999e-05 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583569+0000", 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046012680000000004 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:51.583628+0000", 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8473000000000001e-05 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2293 4.9 4:9441befd:::smithi20438004-8:head [write 1586353~512385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 39.145245119999998, 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066015969999999998, 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.509 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2293 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271874+0000", 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.497e-06 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272359+0000", 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048419700000000001 2024-09-13T11:33:34.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272399+0000", 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0754000000000003e-05 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272410+0000", 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272470+0000", 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0603999999999998e-05 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272575+0000", 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104901 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272731+0000", 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015523300000000001 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.274392+0000", 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016616070000000001 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.276887+0000", 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024952360000000001 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278386+0000", 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014986540000000001 2024-09-13T11:33:34.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278439+0000", 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2624999999999998e-05 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278475+0000", 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5812999999999997e-05 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.515 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2494 4.d e63/31 4:b15c1843:::smithi20438004-33:head v 63'65, mlcod=63'62)", 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.768031411000003, 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064248049999999996, 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649088+0000", 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.78e-07 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649096+0000", 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3790000000000006e-06 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649108+0000", 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1751e-05 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649116+0000", 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3850000000000002e-06 2024-09-13T11:33:34.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649307+0000", 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019067199999999999 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649325+0000", 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7859e-05 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655462+0000", 2024-09-13T11:33:34.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061369980000000003 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655511+0000", 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9883000000000003e-05 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.4.22:41 3.1 e63/51 3:b93f8350:::9968f6d703223c53b55081382c31f0a72d88ed4f88670426e6b8ab5df371e41b:head v 63'230, mlcod=63'227)", 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.315661925999997, 2024-09-13T11:33:34.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063303989999999996, 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:34.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101457+0000", 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101463+0000", 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7529999999999998e-06 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101473+0000", 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0154e-05 2024-09-13T11:33:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101483+0000", 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101542+0000", 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9132999999999998e-05 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101560+0000", 2024-09-13T11:33:34.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7156e-05 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107733+0000", 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061735740000000002 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107787+0000", 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3389e-05 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3068 3.1 e72/51 3:ba0ff2fc:::35d746116e015b742d2355e1d921a21f28a06eaf655b8cfcf0782553e2cf9a5a:head v 72'267, mlcod=70'265)", 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.800508549, 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073641619999999996, 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.526 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620678+0000", 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040678880000000004 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620694+0000", 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.611e-05 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620705+0000", 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1124999999999999e-05 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620768+0000", 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2626000000000007e-05 2024-09-13T11:33:34.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620794+0000", 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6446000000000001e-05 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623920+0000", 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003126298 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623974+0000", 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3375000000000003e-05 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3600 4.9 4:94e25a0d:::smithi20438004-3:head [write 527819~624406,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.266009081, 2024-09-13T11:33:34.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076023949999999996, 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3600 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151112+0000", 2024-09-13T11:33:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7070000000000001e-06 2024-09-13T11:33:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151819+0000", 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00070706700000000003 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151834+0000", 2024-09-13T11:33:34.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4944e-05 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151841+0000", 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1760000000000004e-06 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151897+0000", 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5948e-05 2024-09-13T11:33:34.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151966+0000", 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9283999999999996e-05 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.152076+0000", 2024-09-13T11:33:34.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109633 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154405+0000", 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023291950000000001 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154749+0000", 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034383100000000001 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158638+0000", 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038894250000000002 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158677+0000", 2024-09-13T11:33:34.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9167e-05 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158711+0000", 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4017999999999999e-05 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3987 4.b e81/52 4:da92e835:::smithi20438004-50:head v 81'69, mlcod=76'67)", 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:34.540 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.20765382800000001, 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068559900000000002, 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:34.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7100000000000001e-07 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210320+0000", 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085537199999999997 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210332+0000", 2024-09-13T11:33:34.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.224e-05 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210343+0000", 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0549e-05 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210424+0000", 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1209000000000001e-05 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210450+0000", 2024-09-13T11:33:34.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6673e-05 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216271+0000", 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058204060000000002 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216320+0000", 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9370000000000003e-05 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.545 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:34.546 INFO:tasks.rados.rados.0.smithi204.stdout:1681: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:34.546 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:34.545+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:34.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:34.610 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:34.646 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:34.645+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:34.653 INFO:tasks.rados.rados.0.smithi204.stdout:1683: finishing write tid 1 to smithi20438004-50 2024-09-13T11:33:34.653 INFO:tasks.rados.rados.0.smithi204.stdout:1683: finishing write tid 2 to smithi20438004-50 2024-09-13T11:33:34.653 INFO:tasks.rados.rados.0.smithi204.stdout:1685: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-13T11:33:34.660 INFO:tasks.rados.rados.0.smithi204.stdout:1685: writing smithi20438004-22 from 716064 to 1355950 tid 1 2024-09-13T11:33:34.665 INFO:tasks.rados.rados.0.smithi204.stdout:1685: writing smithi20438004-22 from 1815191 to 2315640 tid 2 2024-09-13T11:33:34.671 INFO:tasks.rados.rados.0.smithi204.stdout:1685: writing smithi20438004-22 from 2817011 to 3343936 tid 3 2024-09-13T11:33:34.672 INFO:tasks.rados.rados.0.smithi204.stdout:1680: done (5 left) 2024-09-13T11:33:34.672 INFO:tasks.rados.rados.0.smithi204.stdout:1681: done (4 left) 2024-09-13T11:33:34.672 INFO:tasks.rados.rados.0.smithi204.stdout:1682: done (3 left) 2024-09-13T11:33:34.672 INFO:tasks.rados.rados.0.smithi204.stdout:1686: set_chunk oid 34 target oid 2024-09-13T11:33:34.680 INFO:tasks.rados.rados.0.smithi204.stdout:1686: set_chunk oid 34 offset: 2279936 length: 374784 target oid 1460247bb79406a30e10c70ae58ef5cf200f5e8c2a4c626872cae3900539fd29 offset: 0 snap -1 2024-09-13T11:33:34.680 INFO:tasks.rados.rados.0.smithi204.stdout:1687: tier_flush oid 32 2024-09-13T11:33:34.680 INFO:tasks.rados.rados.0.smithi204.stdout:1687: tier_flush oid 32 snap -1 2024-09-13T11:33:34.680 INFO:tasks.rados.rados.0.smithi204.stdout:1688: read oid 44 snap -1 2024-09-13T11:33:34.680 INFO:tasks.rados.rados.0.smithi204.stdout:1688: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-13T11:33:34.680 INFO:tasks.rados.rados.0.smithi204.stdout:1689: set_chunk oid 49 target oid 2024-09-13T11:33:34.694 INFO:tasks.rados.rados.0.smithi204.stdout:1689: set_chunk oid 49 offset: 455168 length: 1846784 target oid 071a50f94fe8d2fa51f8eabefe5926235717ace071b6aac2e9abc7a021b436fc offset: 0 snap -1 2024-09-13T11:33:34.694 INFO:tasks.rados.rados.0.smithi204.stdout:1690: tier_flush oid 27 2024-09-13T11:33:34.694 INFO:tasks.rados.rados.0.smithi204.stdout:1690: tier_flush oid 27 snap -1 2024-09-13T11:33:34.695 INFO:tasks.rados.rados.0.smithi204.stdout:1683: finishing write tid 3 to smithi20438004-50 2024-09-13T11:33:34.695 INFO:tasks.rados.rados.0.smithi204.stdout:1691: tier_flush oid 8 2024-09-13T11:33:34.695 INFO:tasks.rados.rados.0.smithi204.stdout:1691: tier_flush oid 8 snap -1 2024-09-13T11:33:34.695 INFO:tasks.rados.rados.0.smithi204.stdout:1692: set_chunk oid 39 target oid 2024-09-13T11:33:34.699 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:34.699 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:34.699 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:34.705 INFO:tasks.rados.rados.0.smithi204.stdout:1692: set_chunk oid 39 offset: 2179072 length: 840704 target oid 57291c528df4517ebbc7d9e2882eaa8737a117e06572fdc4c24e0347fff34965 offset: 0 snap -1 2024-09-13T11:33:34.705 INFO:tasks.rados.rados.0.smithi204.stdout:1693: tier_promote oid 5 2024-09-13T11:33:34.705 INFO:tasks.rados.rados.0.smithi204.stdout:1694: tier_evict oid 4 2024-09-13T11:33:34.705 INFO:tasks.rados.rados.0.smithi204.stdout:1694: tier_evict oid 4 snap -1 2024-09-13T11:33:34.705 INFO:tasks.rados.rados.0.smithi204.stdout:1695: read oid 15 snap -1 2024-09-13T11:33:34.705 INFO:tasks.rados.rados.0.smithi204.stdout:1695: expect (ObjNum 301 snap 0 seq_num 301) 2024-09-13T11:33:34.705 INFO:tasks.rados.rados.0.smithi204.stdout:1696: read oid 1 snap -1 2024-09-13T11:33:34.706 INFO:tasks.rados.rados.0.smithi204.stdout:1696: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-13T11:33:34.706 INFO:tasks.rados.rados.0.smithi204.stdout:1697: write oid 31 current snap is 0 2024-09-13T11:33:34.706 INFO:tasks.rados.rados.0.smithi204.stdout:1697: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-13T11:33:34.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:34.713 INFO:tasks.rados.rados.0.smithi204.stdout:1697: writing smithi20438004-31 from 732871 to 1448006 tid 1 2024-09-13T11:33:34.719 INFO:tasks.rados.rados.0.smithi204.stdout:1697: writing smithi20438004-31 from 1873673 to 2517780 tid 2 2024-09-13T11:33:34.728 INFO:tasks.rados.rados.0.smithi204.stdout:1697: writing smithi20438004-31 from 3244434 to 3932871 tid 3 2024-09-13T11:33:34.728 INFO:tasks.rados.rados.0.smithi204.stdout:1698: read oid 43 snap -1 2024-09-13T11:33:34.728 INFO:tasks.rados.rados.0.smithi204.stdout:1698: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: finishing write tid 4 to smithi20438004-50 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: finishing write tid 5 to smithi20438004-50 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: oid 50 updating version 0 to 72 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: oid 50 version 72 is already newer than 71 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: oid 50 version 72 is already newer than 70 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: oid 50 version 72 is already newer than 69 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 72 (ObjNum 350 snap 0 seq_num 350) dirty exists 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: left oid 50 (ObjNum 350 snap 0 seq_num 350) 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1684: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1684: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1683: done (15 left) 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1699: delete oid 40 current snap is 0 2024-09-13T11:33:34.729 INFO:tasks.rados.rados.0.smithi204.stdout:1684: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1684: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1684: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1684: oid 38 updating version 0 to 142 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1684: oid 38 version 142 is already newer than 139 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1684: oid 38 version 142 is already newer than 141 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1684: oid 38 version 142 is already newer than 140 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 142 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1684: left oid 38 (ObjNum 351 snap 0 seq_num 351) 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1685: finishing write tid 1 to smithi20438004-22 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1685: finishing write tid 2 to smithi20438004-22 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1685: finishing write tid 3 to smithi20438004-22 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1685: finishing write tid 4 to smithi20438004-22 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1685: finishing write tid 5 to smithi20438004-22 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1685: oid 22 updating version 0 to 107 2024-09-13T11:33:34.730 INFO:tasks.rados.rados.0.smithi204.stdout:1685: oid 22 updating version 107 to 108 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:1685: oid 22 updating version 108 to 109 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:1685: oid 22 version 109 is already newer than 106 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 109 (ObjNum 352 snap 0 seq_num 352) dirty exists 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:1685: left oid 22 (ObjNum 352 snap 0 seq_num 352) 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:1686: finishing set_chunk to oid 34 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 134 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:1688: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-13T11:33:34.731 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:34.748 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:34.746+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:34.820 INFO:tasks.rados.rados.0.smithi204.stdout:1689: finishing set_chunk to oid 49 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 455168 length: 1846784 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:1690: got (0) Success 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 102 (ObjNum 302 snap 0 seq_num 302) dirty exists 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:1687: got (0) Success 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:1694: got (22) Invalid argument 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:1693: got (0) Success 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 83 (ObjNum 343 snap 0 seq_num 343) dirty exists 2024-09-13T11:33:34.821 INFO:tasks.rados.rados.0.smithi204.stdout:1684: done (15 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1685: done (14 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1686: done (13 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1687: done (12 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1688: done (11 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1689: done (10 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1690: done (9 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1693: done (8 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1694: done (7 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1699: done (6 left) 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1700: read oid 29 snap -1 2024-09-13T11:33:34.822 INFO:tasks.rados.rados.0.smithi204.stdout:1700: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:1701: read oid 6 snap -1 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:1701: expect (ObjNum 347 snap 0 seq_num 347) 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:1702: delete oid 24 current snap is 0 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:1692: finishing set_chunk to oid 39 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 112 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:1691: got (0) Success 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-13T11:33:34.823 INFO:tasks.rados.rados.0.smithi204.stdout:1696: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-13T11:33:34.849 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:34.847+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:34.858 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:34.858 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:34.858 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:34.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:34.913 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:34.950 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:34.949+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:34.967 INFO:tasks.rados.rados.0.smithi204.stdout:1695: expect (ObjNum 301 snap 0 seq_num 301) 2024-09-13T11:33:34.973 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1102592 2024-09-13T11:33:34.974 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.975 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 68.449618246, 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:33:34.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:33:34.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:33:34.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.760680105000006, 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:33:34.985 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:33:34.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:33:34.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:33:34.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.752337906999998, 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:34.992 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:34.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:34.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:33:34.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.731979903999999, 2024-09-13T11:33:34.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:33:35.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:33:35.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:35.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.724179144999994, 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.007 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:35.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:33:35.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.174672337000004, 2024-09-13T11:33:35.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:33:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:33:35.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:33:35.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:797 4.8 4:1cf41451:::smithi20438004-7:head [write 2778819~95145] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:35.019 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.161309774000003, 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086937119999999993, 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 797 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:35.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810972+0000", 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810974+0000", 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7010000000000001e-06 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811111+0000", 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013712699999999999 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811123+0000", 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1854e-05 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811131+0000", 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3979999999999999e-06 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813131+0000", 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019997740000000002 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813239+0000", 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108378 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813421+0000", 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018227099999999999 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814991+0000", 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015690599999999999 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816465+0000", 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001474784 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819575+0000", 2024-09-13T11:33:35.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031099769999999999 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819626+0000", 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0609999999999998e-05 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819666+0000", 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9777999999999998e-05 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:798 4.8 4:1cf41451:::smithi20438004-7:head [setxattr _header (55) in=7b,truncate 2873964] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.161136147999997, 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087603679999999993, 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.026 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 798 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811146+0000", 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811150+0000", 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8770000000000003e-06 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811157+0000", 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3799999999999999e-06 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811163+0000", 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813998+0000", 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002835382 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814069+0000", 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1099000000000005e-05 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814212+0000", 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142647 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816475+0000", 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022629799999999999 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.816520+0000", 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5099999999999998e-05 2024-09-13T11:33:35.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819739+0000", 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032186570000000002 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819820+0000", 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1527999999999996e-05 2024-09-13T11:33:35.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819906+0000", 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5845000000000002e-05 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:799 4.8 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.161094808000001, 2024-09-13T11:33:35.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089993910000000007, 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 799 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811187+0000", 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811188+0000", 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9599999999999999e-07 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811192+0000", 2024-09-13T11:33:35.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.566e-06 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811202+0000", 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4790000000000002e-06 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.811209+0000", 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2030000000000003e-06 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814715+0000", 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035061150000000002 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.814798+0000", 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2854000000000003e-05 2024-09-13T11:33:35.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.819962+0000", 2024-09-13T11:33:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051643419999999997 2024-09-13T11:33:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820067+0000", 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010525 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.820186+0000", 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118886 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.039 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.212483801999994, 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:33:35.040 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:33:35.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:33:35.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:33:35.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1485 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~3966419,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 57.862016705999999, 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01527358, 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1485 2024-09-13T11:33:35.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:35.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2300000000000002e-07 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110269+0000", 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110274+0000", 2024-09-13T11:33:35.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2349999999999999e-06 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110278+0000", 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6100000000000002e-06 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117718+0000", 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074403120000000001 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117821+0000", 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103501 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.125539+0000", 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077171590000000003 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1636 4.8 4:1d367c64:::smithi20438004-45:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:35.052 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 53.971849538999997, 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091050230000000003, 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1636 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000433+0000", 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1200000000000003e-07 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000436+0000", 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000443+0000", 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8619999999999997e-06 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000447+0000", 2024-09-13T11:33:35.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8800000000000001e-06 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.001976+0000", 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001529774 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.002101+0000", 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124824 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009537+0000", 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074360850000000003 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1637 4.8 4:1d367c64:::smithi20438004-45:head [sparse-read 0~2368645] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 53.971827009999998, 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010163363, 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1637 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:35.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000457+0000", 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000460+0000", 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3720000000000001e-06 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000463+0000", 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.225e-06 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009568+0000", 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091047630000000001 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009626+0000", 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8142e-05 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.010618+0000", 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099181000000000009 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1749 4.8 4:1d367c64:::smithi20438004-45:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.184580844000003, 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49694346099999998, 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1749 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787703+0000", 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6840000000000001e-06 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787708+0000", 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787721+0000", 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2405000000000001e-05 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787728+0000", 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.391e-06 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787780+0000", 2024-09-13T11:33:35.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1897000000000001e-05 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787845+0000", 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5184000000000005e-05 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284644+0000", 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49679929099999998 2024-09-13T11:33:35.066 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3085 4.d 4:b15c1843:::smithi20438004-33:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e72)", 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.697705423999999, 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012088290999999999, 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.067 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3085 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274578+0000", 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0609999999999998e-06 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274588+0000", 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9119999999999993e-06 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274602+0000", 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3655000000000001e-05 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274613+0000", 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1199000000000001e-05 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.275921+0000", 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013075000000000001 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:33:35.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.275933+0000", 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1905999999999999e-05 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280509+0000", 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045760560000000002 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280683+0000", 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017461199999999999 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280858+0000", 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017475999999999999 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.284147+0000", 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032893900000000001 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.284277+0000", 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012992 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286340+0000", 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020622779999999999 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286614+0000", 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274486 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286665+0000", 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0556000000000002e-05 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3500 4.d 4:bca378c8:::smithi20438004-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.213345, 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026796718000000001, 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3500 2024-09-13T11:33:35.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758939+0000", 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758952+0000", 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2337000000000001e-05 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758985+0000", 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3099000000000003e-05 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759005+0000", 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0197e-05 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759114+0000", 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010864799999999999 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759208+0000", 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4728999999999994e-05 2024-09-13T11:33:35.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785733+0000", 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026525014999999999 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3505 4.14 4:2c957d44:::smithi20438004-22:head [write 1940712~656106] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.196950206, 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010057587999999999, 2024-09-13T11:33:35.080 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3505 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775333+0000", 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.488e-06 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776349+0000", 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010159909999999999 2024-09-13T11:33:35.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776360+0000", 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1481000000000001e-05 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776369+0000", 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776417+0000", 2024-09-13T11:33:35.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8182e-05 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776474+0000", 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6669000000000002e-05 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776581+0000", 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010673199999999999 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.778281+0000", 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001700811 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785203+0000", 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069212960000000004 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785249+0000", 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6448e-05 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785290+0000", 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1340000000000001e-05 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785389+0000", 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8607e-05 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3568 4.d 4:b15c1843:::smithi20438004-33:head [write 477398~798659,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:35.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.6871397209999994, 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010217238999999999, 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3568 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285148+0000", 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8449999999999999e-06 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286131+0000", 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00098294599999999995 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286146+0000", 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4712000000000001e-05 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286153+0000", 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6399999999999997e-06 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286206+0000", 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3284e-05 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286271+0000", 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286382+0000", 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011042799999999999 2024-09-13T11:33:35.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.288353+0000", 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019708669999999998 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.288371+0000", 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8538000000000001e-05 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295281+0000", 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069098720000000001 2024-09-13T11:33:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295321+0000", 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0145000000000003e-05 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295359+0000", 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8040000000000002e-05 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3632 4.8 4:1251e6cc:::smithi20438004-13:head [read 618102~2928175,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7413056069999993, 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010153435000000001, 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3632 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.093 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230977+0000", 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9200000000000001e-07 2024-09-13T11:33:35.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230984+0000", 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8990000000000004e-06 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230991+0000", 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5089999999999996e-06 2024-09-13T11:33:35.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230998+0000", 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3999999999999997e-06 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.235260+0000", 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042621330000000004 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.235319+0000", 2024-09-13T11:33:35.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9549999999999997e-05 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.241130+0000", 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058103520000000004 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.097 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3839 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:33:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.3996641189999997, 2024-09-13T11:33:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023869665000000002, 2024-09-13T11:33:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.098 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3839 2024-09-13T11:33:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572620+0000", 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0339999999999999e-06 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572630+0000", 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0056e-05 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572658+0000", 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8756e-05 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572673+0000", 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572789+0000", 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011540900000000001 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572850+0000", 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1397999999999997e-05 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.596487+0000", 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023637162 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.105 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:35.107 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:35.105+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 222067 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:1697: finishing write tid 1 to smithi20438004-31 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:1697: finishing write tid 2 to smithi20438004-31 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:1697: finishing write tid 3 to smithi20438004-31 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:1697: finishing write tid 4 to smithi20438004-31 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:1697: finishing write tid 5 to smithi20438004-31 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:1697: oid 31 updating version 0 to 57 2024-09-13T11:33:35.107 INFO:tasks.rados.rados.0.smithi204.stdout:1697: oid 31 version 57 is already newer than 56 2024-09-13T11:33:35.108 INFO:tasks.rados.rados.0.smithi204.stdout:1697: oid 31 updating version 57 to 58 2024-09-13T11:33:35.108 INFO:tasks.rados.rados.0.smithi204.stdout:1697: oid 31 updating version 58 to 59 2024-09-13T11:33:35.108 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 59 (ObjNum 353 snap 0 seq_num 353) dirty exists 2024-09-13T11:33:35.108 INFO:tasks.rados.rados.0.smithi204.stdout:1697: left oid 31 (ObjNum 353 snap 0 seq_num 353) 2024-09-13T11:33:35.108 INFO:tasks.rados.rados.0.smithi204.stdout:1698: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:35.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:35.175 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:35.208 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:35.206+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:35.217 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:35.217 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:35.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:35.308 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:35.307+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:35.322 INFO:tasks.rados.rados.0.smithi204.stdout:1700: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:35.332 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:35.332 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:35.332 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:35.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:35.364 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3181995 2024-09-13T11:33:35.409 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:35.408+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:35.422 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.040217910000003, 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028959070000000001, 2024-09-13T11:33:35.451 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:35.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408205+0000", 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409400+0000", 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011945110000000001 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409410+0000", 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0159999999999999e-05 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409416+0000", 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409467+0000", 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1270999999999998e-05 2024-09-13T11:33:35.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409489+0000", 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1682999999999999e-05 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411056+0000", 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015677569999999999 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411100+0000", 2024-09-13T11:33:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4036999999999999e-05 2024-09-13T11:33:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2315 4.0 e58/56 4:001d90c2:::smithi20438004-28:head v 58'53, mlcod=57'51)", 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 39.328760101, 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031817899999999999, 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119664+0000", 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8330000000000001e-06 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120926+0000", 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012617710000000001 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120940+0000", 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4544e-05 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120952+0000", 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1676e-05 2024-09-13T11:33:35.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121050+0000", 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8177999999999994e-05 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121071+0000", 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1174000000000001e-05 2024-09-13T11:33:35.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122794+0000", 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00172215 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122844+0000", 2024-09-13T11:33:35.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0464000000000003e-05 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4b088bf8:::smithi20438004-24:head, version: 60'85, data_included: [540672~892928,1630208~155648], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4b088bf8:::smithi20438004-24:head@60'85, size: 3058874, copy_subset: [(0, 3058874)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1785856, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.135232518999999, 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003437201, 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.463 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313191+0000", 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.046e-06 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315519+0000", 2024-09-13T11:33:35.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023280670000000001 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315551+0000", 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2440000000000001e-05 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315574+0000", 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2286e-05 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315668+0000", 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929000000000001e-05 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315679+0000", 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1e-05 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.316627+0000", 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094843300000000005 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.468 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2354 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'52, mlcod=57'50)", 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.127236369000002, 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058622930000000002, 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.469 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321187+0000", 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2200000000000003e-07 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323519+0000", 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002331891 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323532+0000", 2024-09-13T11:33:35.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3494e-05 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323542+0000", 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323612+0000", 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9809999999999997e-05 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323638+0000", 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6526999999999999e-05 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.326990+0000", 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033519270000000002 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327048+0000", 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8158999999999997e-05 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.473 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2355 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'53, mlcod=57'50)", 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.124187994000003, 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028910490000000001, 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324234+0000", 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324235+0000", 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6699999999999999e-07 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324282+0000", 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7437e-05 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324290+0000", 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3170000000000002e-06 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324297+0000", 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.691e-06 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324944+0000", 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064681799999999996 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324977+0000", 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2591e-05 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327083+0000", 2024-09-13T11:33:35.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021065540000000001 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327125+0000", 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2274000000000002e-05 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2356 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'54, mlcod=57'50)", 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:35.478 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.124114329000001, 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030409059999999999, 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:35.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0100000000000001e-07 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324312+0000", 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324318+0000", 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6119999999999998e-06 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324323+0000", 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325363+0000", 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001040116 2024-09-13T11:33:35.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325383+0000", 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9644999999999999e-05 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327310+0000", 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001927205 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327349+0000", 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8930000000000002e-05 2024-09-13T11:33:35.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2513 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'86, mlcod=0'0)", 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.382592340999999, 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034473500000000001, 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8099999999999999e-07 2024-09-13T11:33:35.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066800+0000", 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096952400000000004 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066816+0000", 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6010000000000001e-05 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066826+0000", 2024-09-13T11:33:35.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0679000000000001e-05 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066970+0000", 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014400900000000001 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.067020+0000", 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9595000000000002e-05 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069220+0000", 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022001640000000001 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069277+0000", 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7188000000000002e-05 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2515 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'88, mlcod=63'86)", 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:35.487 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.369377733999997, 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007260702, 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4300000000000002e-07 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080302+0000", 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001256889 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080314+0000", 2024-09-13T11:33:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.134e-05 2024-09-13T11:33:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080319+0000", 2024-09-13T11:33:35.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6699999999999999e-06 2024-09-13T11:33:35.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080371+0000", 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2039000000000001e-05 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080397+0000", 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5996e-05 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086251+0000", 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058537789999999999 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086305+0000", 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4146e-05 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2516 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'89, mlcod=63'86)", 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:35.493 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.368001554000003, 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062074870000000002, 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7199999999999999e-07 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080426+0000", 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080434+0000", 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2930000000000001e-06 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080443+0000", 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3699999999999995e-06 2024-09-13T11:33:35.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081319+0000", 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087608100000000002 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081346+0000", 2024-09-13T11:33:35.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7282999999999999e-05 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086595+0000", 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052492759999999998 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086628+0000", 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2836999999999997e-05 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2718 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'58, mlcod=58'56)", 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.324602801000001, 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039829779999999999, 2024-09-13T11:33:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125730+0000", 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001910213 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125744+0000", 2024-09-13T11:33:35.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3326000000000001e-05 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125752+0000", 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125811+0000", 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9582e-05 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125836+0000", 2024-09-13T11:33:35.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4482e-05 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127747+0000", 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019112599999999999 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127803+0000", 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5633000000000001e-05 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2719 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'59, mlcod=60'57)", 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.316825541, 2024-09-13T11:33:35.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032139159999999998, 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131598+0000", 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7999999999999993e-07 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132663+0000", 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010650010000000001 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132690+0000", 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6970000000000001e-05 2024-09-13T11:33:35.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132703+0000", 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.36e-05 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132837+0000", 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013366699999999999 2024-09-13T11:33:35.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132900+0000", 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3331999999999996e-05 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134769+0000", 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018688520000000001 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134811+0000", 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1514000000000003e-05 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3042 4.12 e71/61 4:4b088bf8:::smithi20438004-24:head v 71'99, mlcod=70'97)", 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.332158894999999, 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036713660000000001, 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.510 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116264+0000", 2024-09-13T11:33:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8300000000000001e-07 2024-09-13T11:33:35.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117477+0000", 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012129280000000001 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117486+0000", 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7100000000000002e-06 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117496+0000", 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6940000000000007e-06 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117579+0000", 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3207999999999993e-05 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117606+0000", 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6622999999999999e-05 2024-09-13T11:33:35.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119900+0000", 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022940899999999999 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119935+0000", 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4929999999999999e-05 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3122 4.12 e73/61 4:4b088bf8:::smithi20438004-24:head v 73'102, mlcod=71'100)", 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.072475753999999, 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031004679999999999, 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:35.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.74e-07 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377229+0000", 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001282245 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377237+0000", 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6939999999999995e-06 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377242+0000", 2024-09-13T11:33:35.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377289+0000", 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6981999999999998e-05 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377315+0000", 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6604000000000001e-05 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.379013+0000", 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016974959999999999 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.379047+0000", 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4156000000000003e-05 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.520 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3247 4.0 e74/56 4:04aeeab9:::smithi20438004-5:head v 74'72, mlcod=74'70)", 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.396418+0000", 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.052004739000001, 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028934479999999999, 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.396418+0000", 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.396418+0000", 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.396419+0000", 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5239999999999999e-06 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397566+0000", 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011464260000000001 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397584+0000", 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8138000000000002e-05 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397593+0000", 2024-09-13T11:33:35.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4939999999999992e-06 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397655+0000", 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1298999999999997e-05 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.397674+0000", 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8998e-05 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.399268+0000", 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015941289999999999 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.399311+0000", 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3439999999999997e-05 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.525 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3248 4.0 e74/56 4:04aeeab9:::smithi20438004-5:head v 74'73, mlcod=74'71)", 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.044618286999999, 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031644770000000002, 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403806+0000", 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6500000000000001e-06 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405022+0000", 2024-09-13T11:33:35.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012166569999999999 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405039+0000", 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6942000000000001e-05 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405050+0000", 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0227000000000001e-05 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405112+0000", 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2916999999999994e-05 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405139+0000", 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6594e-05 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.406922+0000", 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017830540000000001 2024-09-13T11:33:35.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.406969+0000", 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6436000000000001e-05 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3286 4.0 e74/56 4:001d90c2:::smithi20438004-28:head v 74'75, mlcod=74'73)", 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.853333902999999, 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048258800000000003, 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595090+0000", 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5319999999999999e-06 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596361+0000", 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001270953 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596378+0000", 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7363999999999998e-05 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596386+0000", 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9640000000000003e-06 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596440+0000", 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3544000000000001e-05 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596459+0000", 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8864000000000001e-05 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.599857+0000", 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033986609999999999 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.599914+0000", 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:35.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3492 4.12 e76/61 4:4b088bf8:::smithi20438004-24:head v 76'108, mlcod=73'106)", 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.701531223, 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028628780000000001, 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.536 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.746891+0000", 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.85e-07 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748263+0000", 2024-09-13T11:33:35.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001371833 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748273+0000", 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0033e-05 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748280+0000", 2024-09-13T11:33:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3829999999999997e-06 2024-09-13T11:33:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748349+0000", 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9832000000000005e-05 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.748368+0000", 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8368000000000001e-05 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.749720+0000", 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001352341 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.749754+0000", 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3902999999999997e-05 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3642 4.12 e76/61 4:4d4c8512:::smithi20438004-29:head v 76'111, mlcod=76'109)", 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2028099619999999, 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003165579, 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245615+0000", 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246791+0000", 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001175773 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246805+0000", 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.401e-05 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246813+0000", 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7100000000000007e-06 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246879+0000", 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5889999999999994e-05 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246909+0000", 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9937000000000002e-05 2024-09-13T11:33:35.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.248741+0000", 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018326709999999999 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.248778+0000", 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6738e-05 2024-09-13T11:33:35.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3846 4.12 e80/61 4:4ba9f189:::smithi20438004-14:head v 80'119, mlcod=78'117)", 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:30.589682+0000", 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.858740053, 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003140809, 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.589682+0000", 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.589682+0000", 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.589683+0000", 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0900000000000002e-07 2024-09-13T11:33:35.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591068+0000", 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001384786 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591081+0000", 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3309000000000001e-05 2024-09-13T11:33:35.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591091+0000", 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.066e-05 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591165+0000", 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3811000000000002e-05 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591186+0000", 2024-09-13T11:33:35.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0718999999999999e-05 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.592771+0000", 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015846409999999999 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.592823+0000", 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2473999999999999e-05 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3847 4.12 e80/61 4:4ba9f189:::smithi20438004-14:head v 80'120, mlcod=78'117)", 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:30.595982+0000", 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8524403769999997, 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029379480000000001, 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:35.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.595982+0000", 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.595982+0000", 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.595983+0000", 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.61e-07 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597086+0000", 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011028769999999999 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597098+0000", 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2595999999999999e-05 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597106+0000", 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7649999999999993e-06 2024-09-13T11:33:35.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597185+0000", 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8542999999999997e-05 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597214+0000", 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9068999999999999e-05 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:35.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.598876+0000", 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016628859999999999 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.598920+0000", 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3451000000000001e-05 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:35.557 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:35.559 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2145157 2024-09-13T11:33:35.559 INFO:tasks.rados.rados.0.smithi204.stdout:1701: expect (ObjNum 347 snap 0 seq_num 347) 2024-09-13T11:33:35.559 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:35.559 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1275905 2024-09-13T11:33:35.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:35.560 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:35.557+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:35.589 INFO:tasks.rados.rados.0.smithi204.stdout:1691: done (8 left) 2024-09-13T11:33:35.589 INFO:tasks.rados.rados.0.smithi204.stdout:1692: done (7 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1695: done (6 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1696: done (5 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1697: done (4 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1698: done (3 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1700: done (2 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1701: done (1 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1702: done (0 left) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1703: read oid 19 snap -1 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1703: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1704: set_chunk oid 42 target oid 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1704: set_chunk oid 42 offset: 0 length: 4000000 target oid smithi20438004- offset: 0 snap -1 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1705: delete oid 37 current snap is 0 2024-09-13T11:33:35.590 INFO:tasks.rados.rados.0.smithi204.stdout:1704: finishing set_chunk to oid 42 2024-09-13T11:33:35.591 INFO:tasks.rados.rados.0.smithi204.stdout:1704: got expected ENOENT (src dne) 2024-09-13T11:33:35.591 INFO:tasks.rados.rados.0.smithi204.stdout:1704: done (2 left) 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1705: done (1 left) 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1706: read oid 38 snap -1 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1706: expect (ObjNum 351 snap 0 seq_num 351) 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1707: read oid 34 snap -1 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1707: expect (ObjNum 340 snap 0 seq_num 340) 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1708: read oid 21 snap -1 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1708: expect deleted 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1709: write (excl) oid 16 current snap is 0 2024-09-13T11:33:35.592 INFO:tasks.rados.rados.0.smithi204.stdout:1709: seq_num 357 ranges {400099=616131,1591248=408851} 2024-09-13T11:33:35.598 INFO:tasks.rados.rados.0.smithi204.stdout:1709: writing smithi20438004-16 from 400099 to 1016230 tid 1 2024-09-13T11:33:35.603 INFO:tasks.rados.rados.0.smithi204.stdout:1709: writing smithi20438004-16 from 1591248 to 2000099 tid 2 2024-09-13T11:33:35.604 INFO:tasks.rados.rados.0.smithi204.stdout:1710: write oid 37 current snap is 0 2024-09-13T11:33:35.604 INFO:tasks.rados.rados.0.smithi204.stdout:1710: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-13T11:33:35.608 INFO:tasks.rados.rados.0.smithi204.stdout:1710: writing smithi20438004-37 from 416906 to 824639 tid 1 2024-09-13T11:33:35.615 INFO:tasks.rados.rados.0.smithi204.stdout:1710: writing smithi20438004-37 from 1366083 to 1877539 tid 2 2024-09-13T11:33:35.615 INFO:tasks.rados.rados.0.smithi204.stdout:1711: tier_flush oid 26 2024-09-13T11:33:35.615 INFO:tasks.rados.rados.0.smithi204.stdout:1711: tier_flush oid 26 snap -1 2024-09-13T11:33:35.615 INFO:tasks.rados.rados.0.smithi204.stdout:1712: read oid 2 snap -1 2024-09-13T11:33:35.615 INFO:tasks.rados.rados.0.smithi204.stdout:1712: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-13T11:33:35.615 INFO:tasks.rados.rados.0.smithi204.stdout:1703: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:35.636 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1880571 2024-09-13T11:33:35.648 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:35.659 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:35.658+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:35.674 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:35.674 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:35.674 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:35.674 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:35.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:35.726 INFO:tasks.rados.rados.0.smithi204.stdout:1706: expect (ObjNum 351 snap 0 seq_num 351) 2024-09-13T11:33:35.760 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:35.759+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:35.785 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:35.785 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:35.786 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:35.786 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:35.786 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:35.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:35.861 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:35.860+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:35.939 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.846062750000002, 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011006991000000001, 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091279+0000", 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.091e-06 2024-09-13T11:33:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098180+0000", 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069015359999999998 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098194+0000", 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.323e-05 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098202+0000", 2024-09-13T11:33:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2879999999999993e-06 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098267+0000", 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4733999999999994e-05 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098296+0000", 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102225+0000", 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039281940000000003 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102285+0000", 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.789257261000003, 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010863593, 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148085+0000", 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2810000000000001e-06 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153254+0000", 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051698430000000004 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153266+0000", 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1973e-05 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153274+0000", 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153331+0000", 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6492000000000001e-05 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153396+0000", 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5083e-05 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158906+0000", 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055099570000000002 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158947+0000", 2024-09-13T11:33:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:33:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.784044714999993, 2024-09-13T11:33:35.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096623879999999992, 2024-09-13T11:33:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157238+0000", 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039414139999999999 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157253+0000", 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5566e-05 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157260+0000", 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.477e-06 2024-09-13T11:33:35.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157315+0000", 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4857e-05 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157341+0000", 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:35.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162899+0000", 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055583480000000003 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162958+0000", 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9169000000000002e-05 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.774386629999995, 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011410943999999999, 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162955+0000", 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0669999999999999e-06 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167312+0000", 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004357072 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167333+0000", 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0842000000000001e-05 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167341+0000", 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4430000000000003e-06 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167406+0000", 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4245000000000006e-05 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167433+0000", 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7500000000000001e-05 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174280+0000", 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068473969999999999 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174365+0000", 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4377999999999997e-05 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.761558867000005, 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099135219999999993, 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175783+0000", 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179929+0000", 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041461989999999997 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179943+0000", 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4731e-05 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179953+0000", 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180049+0000", 2024-09-13T11:33:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6909000000000003e-05 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180073+0000", 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4077e-05 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185641+0000", 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055674330000000001 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185695+0000", 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4273e-05 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:258 4.3 4:c717f8db:::smithi20438004-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.723791601000002, 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017162947000000001, 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 258 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213551+0000", 2024-09-13T11:33:35.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0760000000000001e-06 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213566+0000", 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5441999999999999e-05 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213585+0000", 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8623000000000001e-05 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213600+0000", 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4783999999999999e-05 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213667+0000", 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213747+0000", 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9794000000000002e-05 2024-09-13T11:33:35.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228488+0000", 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014740964 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229259+0000", 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077144599999999996 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229743+0000", 2024-09-13T11:33:35.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048437000000000003 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230652+0000", 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090904600000000005 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230688+0000", 2024-09-13T11:33:35.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5474e-05 2024-09-13T11:33:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230712+0000", 2024-09-13T11:33:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3954000000000001e-05 2024-09-13T11:33:35.973 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:259 4.1 4:8533c921:::smithi20438004-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:35.974 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.719515998999995, 2024-09-13T11:33:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014548484, 2024-09-13T11:33:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.975 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 259 2024-09-13T11:33:35.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217826+0000", 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6360000000000001e-06 2024-09-13T11:33:35.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217833+0000", 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8700000000000003e-06 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217844+0000", 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217851+0000", 2024-09-13T11:33:35.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217896+0000", 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4808000000000002e-05 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217954+0000", 2024-09-13T11:33:35.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7856000000000003e-05 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229813+0000", 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011859533 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231353+0000", 2024-09-13T11:33:35.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001539503 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232188+0000", 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083500600000000001 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232284+0000", 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6645999999999996e-05 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232325+0000", 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0071999999999999e-05 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232373+0000", 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8498999999999998e-05 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:266 4.4 4:2451796c:::smithi20438004-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.715016203000005, 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099464859999999992, 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 266 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222325+0000", 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222330+0000", 2024-09-13T11:33:35.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999997e-06 2024-09-13T11:33:35.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222341+0000", 2024-09-13T11:33:35.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0528e-05 2024-09-13T11:33:35.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222347+0000", 2024-09-13T11:33:35.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6409999999999996e-06 2024-09-13T11:33:35.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222406+0000", 2024-09-13T11:33:35.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8449000000000002e-05 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222487+0000", 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1167000000000004e-05 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230197+0000", 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077105120000000001 2024-09-13T11:33:35.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230937+0000", 2024-09-13T11:33:35.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000740159 2024-09-13T11:33:35.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231296+0000", 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035862399999999999 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232186+0000", 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089002899999999997 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232234+0000", 2024-09-13T11:33:35.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7414999999999999e-05 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232271+0000", 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7323000000000002e-05 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:283 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.703888067999998, 2024-09-13T11:33:35.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011861159, 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 283 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233453+0000", 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233457+0000", 2024-09-13T11:33:35.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1150000000000004e-06 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233465+0000", 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233471+0000", 2024-09-13T11:33:35.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3450000000000002e-06 2024-09-13T11:33:35.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:35.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233526+0000", 2024-09-13T11:33:35.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4790000000000002e-05 2024-09-13T11:33:35.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233602+0000", 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6253000000000003e-05 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237634+0000", 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040318580000000001 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238163+0000", 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052849899999999998 2024-09-13T11:33:35.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242711+0000", 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045478530000000001 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245235+0000", 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025239160000000002 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245286+0000", 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1739e-05 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245314+0000", 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7280999999999999e-05 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:35.999 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:284 4.1 4:8b052c81:::smithi20438004-35:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.703538055999999, 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011653294, 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 284 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233803+0000", 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4600000000000003e-07 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233808+0000", 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233817+0000", 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9870000000000005e-06 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233824+0000", 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8290000000000002e-06 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233874+0000", 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9877999999999999e-05 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233943+0000", 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:36.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238024+0000", 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040810350000000002 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238440+0000", 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041507799999999999 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242785+0000", 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043457890000000001 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245363+0000", 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025773359999999999 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245416+0000", 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3156999999999999e-05 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245456+0000", 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0000000000000003e-05 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.005 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:286 4.5 4:a9b0c9bd:::smithi20438004-37:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.702556294999994, 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013081485, 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 286 2024-09-13T11:33:36.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234785+0000", 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234790+0000", 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.651e-06 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234800+0000", 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234807+0000", 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.861e-06 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234868+0000", 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0582999999999999e-05 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234967+0000", 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054743730000000003 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241205+0000", 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076331700000000001 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241351+0000", 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619399999999999 2024-09-13T11:33:36.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247766+0000", 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064151290000000003 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247825+0000", 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8739999999999997e-05 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247866+0000", 2024-09-13T11:33:36.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747000000000001e-05 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:290 4.1 4:8d6fdfdb:::smithi20438004-41:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.698002145999993, 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014414148999999999, 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 290 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239339+0000", 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239344+0000", 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2290000000000002e-06 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239354+0000", 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239361+0000", 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.092e-06 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239406+0000", 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126000000000001e-05 2024-09-13T11:33:36.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239452+0000", 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6192000000000001e-05 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248478+0000", 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090262209999999992 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249309+0000", 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083057300000000003 2024-09-13T11:33:36.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253400+0000", 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040908760000000002 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253662+0000", 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026229500000000001 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253715+0000", 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2836e-05 2024-09-13T11:33:36.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253753+0000", 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7526e-05 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:292 4.1 4:84fa1f64:::smithi20438004-43:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.697578933000003, 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010951834000000001, 2024-09-13T11:33:36.018 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 292 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8800000000000002e-07 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239765+0000", 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:33:36.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239773+0000", 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8229999999999995e-06 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239779+0000", 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9900000000000002e-06 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239831+0000", 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2326e-05 2024-09-13T11:33:36.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239910+0000", 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8623999999999998e-05 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245574+0000", 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005663927 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.246099+0000", 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052478300000000002 2024-09-13T11:33:36.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247963+0000", 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001864166 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250625+0000", 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002662228 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250676+0000", 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0989000000000003e-05 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250713+0000", 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7360000000000001e-05 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:317 4.1 4:8c30e60e:::smithi20438004-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.024 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.686455027999997, 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013134860999999999, 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 317 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.025 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:36.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250892+0000", 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250900+0000", 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250908+0000", 2024-09-13T11:33:36.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.337e-06 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250967+0000", 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8201000000000001e-05 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251045+0000", 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299000000000004e-05 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:36.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258769+0000", 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007724323 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259599+0000", 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082958200000000004 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260992+0000", 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00139291 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263918+0000", 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029263190000000001 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263971+0000", 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2670999999999998e-05 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264020+0000", 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-09-13T11:33:36.030 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:330 4.3 4:c7f26bfe:::smithi20438004-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.685232213000006, 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011943621, 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.031 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 330 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252109+0000", 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252116+0000", 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9199999999999998e-06 2024-09-13T11:33:36.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252129+0000", 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2768e-05 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252138+0000", 2024-09-13T11:33:36.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5939999999999994e-06 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252205+0000", 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252280+0000", 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4930000000000003e-05 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259484+0000", 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00720434 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259994+0000", 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051054200000000001 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260333+0000", 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033855000000000002 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263947+0000", 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00361452 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264015+0000", 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7155000000000003e-05 2024-09-13T11:33:36.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264052+0000", 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305000000000003e-05 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:333 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.684942016999997, 2024-09-13T11:33:36.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011814712, 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 333 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252406+0000", 2024-09-13T11:33:36.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252416+0000", 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-05 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252423+0000", 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252486+0000", 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2805999999999995e-05 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252540+0000", 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3897000000000003e-05 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259138+0000", 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065977429999999997 2024-09-13T11:33:36.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259785+0000", 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064727700000000003 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264056+0000", 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042706710000000002 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264152+0000", 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5877000000000004e-05 2024-09-13T11:33:36.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264180+0000", 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8674999999999999e-05 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264213+0000", 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3015000000000002e-05 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.045 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:433 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:36.045 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:36.045 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.660391169999997, 2024-09-13T11:33:36.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098551839999999995, 2024-09-13T11:33:36.045 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.046 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.046 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:36.046 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.046 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.046 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.046 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-13T11:33:36.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.047 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:36.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:36.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276950+0000", 2024-09-13T11:33:36.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999995e-07 2024-09-13T11:33:36.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276955+0000", 2024-09-13T11:33:36.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1059999999999999e-06 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276965+0000", 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276977+0000", 2024-09-13T11:33:36.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1518000000000001e-05 2024-09-13T11:33:36.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277035+0000", 2024-09-13T11:33:36.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8133000000000001e-05 2024-09-13T11:33:36.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277125+0000", 2024-09-13T11:33:36.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0148999999999994e-05 2024-09-13T11:33:36.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:36.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282235+0000", 2024-09-13T11:33:36.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005109461 2024-09-13T11:33:36.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282725+0000", 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049045700000000002 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286634+0000", 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039087929999999998 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286707+0000", 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2692000000000001e-05 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286772+0000", 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5828999999999996e-05 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286805+0000", 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2134000000000001e-05 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.055 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:791 4.5 4:a5f8f9e2:::smithi20438004-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.237508259999998, 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019917450999999999, 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 791 2024-09-13T11:33:36.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699834+0000", 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699840+0000", 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5989999999999998e-06 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699852+0000", 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699861+0000", 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5259999999999993e-06 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699949+0000", 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8083000000000004e-05 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699953+0000", 2024-09-13T11:33:36.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3869999999999997e-06 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701189+0000", 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012354740000000001 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701250+0000", 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1733e-05 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.719750+0000", 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018499504 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1374 4.1 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.857441131999998, 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011338608999999999, 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.061 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1374 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:36.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079900+0000", 2024-09-13T11:33:36.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-07 2024-09-13T11:33:36.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079901+0000", 2024-09-13T11:33:36.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-13T11:33:36.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079905+0000", 2024-09-13T11:33:36.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2039999999999999e-06 2024-09-13T11:33:36.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079908+0000", 2024-09-13T11:33:36.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8200000000000001e-06 2024-09-13T11:33:36.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081697+0000", 2024-09-13T11:33:36.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017893080000000001 2024-09-13T11:33:36.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081802+0000", 2024-09-13T11:33:36.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105231 2024-09-13T11:33:36.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.091238+0000", 2024-09-13T11:33:36.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094355819999999997 2024-09-13T11:33:36.068 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1454 4.1 4:84fa1f64:::smithi20438004-43:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 59.101394288999998, 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025598754000000001, 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:36.069 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1454 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:36.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835947+0000", 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8500000000000002e-07 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835948+0000", 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:36.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835952+0000", 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4259999999999999e-06 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835954+0000", 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8550000000000002e-06 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836019+0000", 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4416000000000001e-05 2024-09-13T11:33:36.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836271+0000", 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025171299999999998 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.861545+0000", 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025274496 2024-09-13T11:33:36.073 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.074 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.074 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.074 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:36.074 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:36.074 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:36.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:36.076 INFO:tasks.rados.rados.0.smithi204.stdout:1707: expect (ObjNum 340 snap 0 seq_num 340) 2024-09-13T11:33:36.076 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1326465 2024-09-13T11:33:36.076 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:36.076 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:36.074+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:36.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:36.118 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:36.176 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:36.174+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:36.179 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:36.179 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:36.179 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:36.179 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:36.188 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1709: finishing write tid 1 to smithi20438004-16 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1710: finishing write tid 1 to smithi20438004-37 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1711: got (2) No such file or directory 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1711: got expected ENOENT (src dne) 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1709: finishing write tid 2 to smithi20438004-16 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1709: finishing write tid 3 to smithi20438004-16 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1709: finishing write tid 4 to smithi20438004-16 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1709: oid 16 updating version 0 to 74 2024-09-13T11:33:36.194 INFO:tasks.rados.rados.0.smithi204.stdout:1709: oid 16 updating version 74 to 75 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1709: oid 16 version 75 is already newer than 73 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 75 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1709: left oid 16 (ObjNum 357 snap 0 seq_num 357) 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1710: finishing write tid 2 to smithi20438004-37 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1710: finishing write tid 3 to smithi20438004-37 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1710: finishing write tid 4 to smithi20438004-37 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1710: oid 37 updating version 0 to 48 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1710: oid 37 updating version 48 to 49 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1710: oid 37 version 49 is already newer than 47 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 49 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1710: left oid 37 (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1703: done (7 left) 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1706: done (6 left) 2024-09-13T11:33:36.195 INFO:tasks.rados.rados.0.smithi204.stdout:1707: done (5 left) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1708: done (4 left) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1709: done (3 left) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1710: done (2 left) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1711: done (1 left) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1713: read oid 10 snap -1 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1713: expect (ObjNum 271 snap 0 seq_num 271) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1714: read oid 7 snap -1 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1714: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1715: tier_promote oid 37 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1716: read oid 41 snap -1 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1716: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1717: read oid 36 snap -1 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1717: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:36.196 INFO:tasks.rados.rados.0.smithi204.stdout:1718: read oid 8 snap -1 2024-09-13T11:33:36.197 INFO:tasks.rados.rados.0.smithi204.stdout:1718: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:36.197 INFO:tasks.rados.rados.0.smithi204.stdout:1719: tier_evict oid 13 2024-09-13T11:33:36.197 INFO:tasks.rados.rados.0.smithi204.stdout:1719: tier_evict oid 13 snap -1 2024-09-13T11:33:36.197 INFO:tasks.rados.rados.0.smithi204.stdout:1720: write oid 20 current snap is 0 2024-09-13T11:33:36.197 INFO:tasks.rados.rados.0.smithi204.stdout:1720: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-13T11:33:36.199 INFO:tasks.rados.rados.0.smithi204.stdout:1720: writing smithi20438004-20 from 433713 to 916695 tid 1 2024-09-13T11:33:36.205 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:36.206 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:36.208 INFO:tasks.rados.rados.0.smithi204.stdout:1720: writing smithi20438004-20 from 1424565 to 2033713 tid 2 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1721: read oid 9 snap -1 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1721: expect (ObjNum 341 snap 0 seq_num 341) 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1722: read oid 17 snap -1 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1722: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1723: read oid 24 snap -1 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1723: expect deleted 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1724: write oid 27 current snap is 0 2024-09-13T11:33:36.209 INFO:tasks.rados.rados.0.smithi204.stdout:1724: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-13T11:33:36.212 INFO:tasks.rados.rados.0.smithi204.stdout:1724: writing smithi20438004-27 from 450520 to 1008751 tid 1 2024-09-13T11:33:36.218 INFO:tasks.rados.rados.0.smithi204.stdout:1724: writing smithi20438004-27 from 1483047 to 1998172 tid 2 2024-09-13T11:33:36.218 INFO:tasks.rados.rados.0.smithi204.stdout:1725: set_chunk oid 47 target oid 2024-09-13T11:33:36.218 INFO:tasks.rados.rados.0.smithi204.stdout:1719: got (22) Invalid argument 2024-09-13T11:33:36.226 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:36.226 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:36.226 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:36.231 INFO:tasks.rados.rados.0.smithi204.stdout:1725: set_chunk oid 47 offset: 1186816 length: 2075648 target oid c53d45399e701f158d165163e47374d7df7ddb1cd4a444ea004239cebe4de822 offset: 0 snap -1 2024-09-13T11:33:36.232 INFO:tasks.rados.rados.0.smithi204.stdout:1719: done (13 left) 2024-09-13T11:33:36.232 INFO:tasks.rados.rados.0.smithi204.stdout:1726: read oid 32 snap -1 2024-09-13T11:33:36.232 INFO:tasks.rados.rados.0.smithi204.stdout:1726: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:36.232 INFO:tasks.rados.rados.0.smithi204.stdout:1727: read oid 39 snap -1 2024-09-13T11:33:36.232 INFO:tasks.rados.rados.0.smithi204.stdout:1727: expect (ObjNum 336 snap 0 seq_num 336) 2024-09-13T11:33:36.232 INFO:tasks.rados.rados.0.smithi204.stdout:1728: set_chunk oid 5 target oid 2024-09-13T11:33:36.243 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:36.243 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:36.245 INFO:tasks.rados.rados.0.smithi204.stdout:1728: set_chunk oid 5 offset: 1605120 length: 1885696 target oid f2e8ef3445892c3133c714bbdf5c6a4b5b1da4a3c79cfe06c2ad422b21e471f5 offset: 0 snap -1 2024-09-13T11:33:36.245 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:36.245 INFO:tasks.rados.rados.0.smithi204.stdout:1714: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-13T11:33:36.259 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:36.259 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:36.276 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:36.276 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:36.277 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:36.275+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:36.293 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:36.293 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:36.310 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:36.310 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:36.326 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:36.332 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:36.332 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:36.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:36.342 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:36.343 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:36.359 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:36.360 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:36.376 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:36.377 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:36.377 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:36.378 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:36.377+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:36.394 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:36.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:36.412 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:36.413 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:36.425 INFO:tasks.rados.rados.0.smithi204.stdout:1715: got (0) Success 2024-09-13T11:33:36.425 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 49 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-09-13T11:33:36.425 INFO:tasks.rados.rados.0.smithi204.stdout:1713: expect (ObjNum 271 snap 0 seq_num 271) 2024-09-13T11:33:36.426 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:36.427 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:36.427 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:36.430 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:36.445 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:36.445 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:36.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:36.497 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:36.496+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:36.506 INFO:tasks.rados.rados.0.smithi204.stdout:1716: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-13T11:33:36.538 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:36.556 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:36.556 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:36.565 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:36.580 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:36.579+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:36.598 INFO:tasks.rados.rados.0.smithi204.stdout:1718: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:36.652 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:36.652 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:36.652 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:36.652 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-13T11:33:36.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-13T11:33:36.681 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:36.680+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:36.709 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:36.709 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:36.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:36.765 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:36.803 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:36.802+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:36.847 INFO:tasks.rados.rados.0.smithi204.stdout:1712: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-13T11:33:36.861 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:36.861 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:36.870 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:36.883 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:36.882+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:36.892 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2611437 2024-09-13T11:33:36.954 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:33:36.955 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:36.955 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 82, 2024-09-13T11:33:36.955 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "b332d022-c5e6-4682-9d3d-2b6994cd0f20", 2024-09-13T11:33:36.955 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-09-13T11:31:50.491934+0000", 2024-09-13T11:33:36.955 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-09-13T11:33:35.417357+0000", 2024-09-13T11:33:36.955 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-09-13T11:32:56.264138+0000", 2024-09-13T11:33:36.955 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-09-13T11:32:44.537150+0000", 2024-09-13T11:33:36.956 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-13T11:33:36.956 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5804032, 2024-09-13T11:33:36.956 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-09-13T11:33:36.956 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-09-13T11:33:36.956 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 7, 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-13T11:33:36.957 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 4, 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 16, 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:02.947125+0000", 2024-09-13T11:33:36.958 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-09-13T11:33:36.959 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-09-13T11:33:36.959 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:36.959 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:36.959 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:36.959 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:36.959 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:36.960 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:36.960 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:36.960 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:36.960 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:36.960 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:36.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:36.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-09-13T11:33:36.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-09-13T11:33:36.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-09-13T11:33:36.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-09-13T11:33:36.962 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-09-13T11:33:36.962 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:36.962 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:36.962 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "25", 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:36.963 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:33:36.964 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:36.964 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:36.964 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-09-13T11:33:36.964 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-09-13T11:33:36.964 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:36.964 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-09-13T11:33:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 15.789999961853027, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 15.789999961853027, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.18999999761581421, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:05.120118+0000", 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "28", 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 28, 2024-09-13T11:33:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 3.9700000286102295, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 3.9700000286102295, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-09-13T11:33:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.5, 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.5, 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "low_tier", 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:17.683275+0000", 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-09-13T11:33:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 4, 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 4, 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 4, 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 4, 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 4, 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "32", 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": {}, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_0", 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-09-13T11:32:18.790550+0000", 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-09-13T11:33:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "is_stretch_pool": false, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 26, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 26, 2024-09-13T11:33:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 26, 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 26, 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 26, 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "58", 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-13T11:33:36.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "56", 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-09-13T11:33:36.987 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-09-13T11:33:36.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-09-13T11:33:36.989 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-09-13T11:33:36.989 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-09-13T11:33:36.989 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-09-13T11:33:36.989 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-09-13T11:33:36.989 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-09-13T11:33:36.989 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-09-13T11:33:36.989 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_cdc_chunk_size": 131072, 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_tier": 3, 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "fingerprint_algorithm": "sha256" 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-09-13T11:33:36.990 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 3.6700000762939453, 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 3.6700000762939453, 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.309999942779541, 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.309999942779541, 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.97729301452636719, 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.98000001907348633, 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.98000001907348633 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:36.991 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "5a4d0d43-6562-441e-925e-4c992ee7b335", 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:36.992 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.994 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2677506544 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.995 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2677506544", 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2677506544", 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2677506544", 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2677506544", 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "80665caa-6933-4f89-b454-128057e9ad92", 2024-09-13T11:33:36.996 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 41, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.997 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6808", 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6809", 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6811", 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:36.999 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6810", 2024-09-13T11:33:37.000 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3561806747 2024-09-13T11:33:37.000 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.000 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.000 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6808/3561806747", 2024-09-13T11:33:37.000 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6809/3561806747", 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6811/3561806747", 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6810/3561806747", 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "eb99fd51-a064-4311-932e-8b6d555409af", 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.001 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 21, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 0, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 41, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6800", 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.003 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.003 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.003 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.003 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.003 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6801", 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6803", 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.004 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6802", 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4217282489 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6800/4217282489", 2024-09-13T11:33:37.005 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6801/4217282489", 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6803/4217282489", 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6802/4217282489", 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "3d840ed9-a410-4bcb-b0f0-70fafdf69842", 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.636688232421875, 2024-09-13T11:33:37.006 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6804", 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6805", 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.008 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6807", 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6806", 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2326505959 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.009 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6804/2326505959", 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6805/2326505959", 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6807/2326505959", 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6806/2326505959", 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0c01b7d7-1309-40fa-913a-97d2a8499293", 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.010 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 72, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.011 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-09-13T11:33:37.013 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3107658120 2024-09-13T11:33:37.014 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/3107658120", 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/3107658120", 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/3107658120", 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/3107658120", 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.015 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "b4cec928-1764-4fe0-8c5d-870d8a0c15c7", 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.016 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 48, 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6800", 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.017 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6801", 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.018 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6803", 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6802", 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3800782738 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6800/3800782738", 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6801/3800782738", 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6803/3800782738", 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6802/3800782738", 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-09-13T11:33:37.020 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "89f6d810-2afc-42b0-b989-052218b6887c", 2024-09-13T11:33:37.021 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.021 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.021 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.021 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.021 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.022 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.022 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.022 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:37.022 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.022 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.022 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.023 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.023 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.023 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6808", 2024-09-13T11:33:37.023 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:37.023 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.024 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.024 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.024 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.024 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6809", 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6811", 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.025 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6810", 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3986362716 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6808/3986362716", 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6809/3986362716", 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6811/3986362716", 2024-09-13T11:33:37.026 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6810/3986362716", 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "ccfee49f-f394-4de0-8e30-853f38e31232", 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.027 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 23, 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6808", 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:37.028 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6809", 2024-09-13T11:33:37.029 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:37.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.033 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.033 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.033 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.033 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.033 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6811", 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.034 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6810", 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1978982202 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6808/1978982202", 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6809/1978982202", 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6811/1978982202", 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6810/1978982202", 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.035 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 8, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "03b1c505-1b92-4daa-9da4-c254be2ba792", 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 41, 2024-09-13T11:33:37.036 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.037 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.038 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3608171775 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/3608171775", 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/3608171775", 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/3608171775", 2024-09-13T11:33:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/3608171775", 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 9, 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "8fbee7a0-2ee6-4551-bf06-4e013e413fb4", 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:37.041 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:37.041 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.041 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.041 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.041 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.042 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 48, 2024-09-13T11:33:37.042 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.042 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.042 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.042 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.043 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.043 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6804", 2024-09-13T11:33:37.043 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:37.043 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.043 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.043 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.044 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.044 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.044 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6805", 2024-09-13T11:33:37.044 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:37.044 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6807", 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:37.045 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6806", 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2580659844 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6804/2580659844", 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6805/2580659844", 2024-09-13T11:33:37.046 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6807/2580659844", 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6806/2580659844", 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 10, 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0808cb41-f62e-49b0-935d-616bea1a4db5", 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.047 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6812", 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:37.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6813", 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.049 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6815", 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.050 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6814", 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2352891695 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6812/2352891695", 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6813/2352891695", 2024-09-13T11:33:37.051 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6815/2352891695", 2024-09-13T11:33:37.052 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6814/2352891695", 2024-09-13T11:33:37.052 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.052 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.052 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.052 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 11, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "427dd574-f2a8-49e5-a32a-cfda023a514a", 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 21, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 72, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.053 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6800", 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.055 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.055 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.055 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.055 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6801", 2024-09-13T11:33:37.055 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:37.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.056 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.056 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.056 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.056 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6803", 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.057 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6802", 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2304836889 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6800/2304836889", 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6801/2304836889", 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6803/2304836889", 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6802/2304836889", 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 12, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "17942689-1130-4b6b-9f98-1605a03731b1", 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 0, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 58, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 59, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.059 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.060 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.061 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.061 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-09-13T11:33:37.061 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:37.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.062 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-09-13T11:33:37.063 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:37.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.063 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.063 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.063 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1688419254 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1688419254", 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1688419254", 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1688419254", 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1688419254", 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.064 INFO:teuthology.orchestra.run.smithi038.stdout: "exists" 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 13, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "62161cab-6d14-4571-971d-217601e38fbc", 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 23, 2024-09-13T11:33:37.065 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 48, 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6812", 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6813", 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6815", 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.067 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.118:6814", 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3727425991 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.118:6812/3727425991", 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.118:6813/3727425991", 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.118:6815/3727425991", 2024-09-13T11:33:37.068 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.118:6814/3727425991", 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 14, 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "559ef141-27db-452d-9577-a6135421f64f", 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.069 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 22, 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 61, 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6804", 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.070 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6805", 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.071 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6807", 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.154:6806", 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 538351706 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.072 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.154:6804/538351706", 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.154:6805/538351706", 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.154:6807/538351706", 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.154:6806/538351706", 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 15, 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "98f9ceba-79df-410a-9788-393057e79c30", 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-09-13T11:33:37.073 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 23, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 72, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6812", 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:37.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6813", 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-09-13T11:33:37.075 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6815", 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.204:6814", 2024-09-13T11:33:37.076 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 501019151 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.204:6812/501019151", 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.204:6813/501019151", 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.204:6815/501019151", 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.204:6814/501019151", 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:37.077 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:03.855267+0000", 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-09-13T11:33:37.078 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:26.046732+0000", 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.079 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:00.198442+0000", 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 41 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:04.310901+0000", 2024-09-13T11:33:37.080 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:03.659118+0000", 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.081 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:09.623455+0000", 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.082 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.808086+0000", 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.499908+0000", 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 8, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:07.828658+0000", 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 9, 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.084 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:22.980747+0000", 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 10, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.085 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:27.377283+0000", 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 11, 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:26.397265+0000", 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.086 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 12, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:32:01.027609+0000", 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 59 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 13, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.087 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.963999+0000", 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 14, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:19.534738+0000", 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 15, 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540719139924082687, 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-09-13T11:33:02.472566+0000", 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-09-13T11:33:37.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [ 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "3.3", 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: 15, 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: 13 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.2", 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: 12, 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: 9 2024-09-13T11:33:37.090 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.4", 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: 8, 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: 5 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.11", 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:37.091 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: 15 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.12", 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: 14, 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: 8, 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: 3 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.13", 2024-09-13T11:33:37.092 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: 14, 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: 6 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.14", 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: 4, 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: 3 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.16", 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: 10, 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: 6, 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: 12 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.19", 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-09-13T11:33:37.094 INFO:teuthology.orchestra.run.smithi038.stdout: 6, 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: 7 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-09-13T11:33:37.095 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-09-13T11:33:37.096 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-09-13T11:33:37.097 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-09-13T11:33:37.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:37.097 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:37.098 INFO:tasks.rados.rados.0.smithi204.stdout:1722: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:37.098 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2763189 2024-09-13T11:33:37.099 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:37.099 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:37.099 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:37.100 INFO:tasks.thrashosds.thrasher:j is {'epoch': 82, 'fsid': 'b332d022-c5e6-4682-9d3d-2b6994cd0f20', 'created': '2024-09-13T11:31:50.491934+0000', 'modified': '2024-09-13T11:33:35.417357+0000', 'last_up_change': '2024-09-13T11:32:56.264138+0000', 'last_in_change': '2024-09-13T11:32:44.537150+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T11:32:02.947125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.789999961853027, 'score_stable': 15.789999961853027, 'optimal_score': 0.1899999976158142, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T11:32:05.120118+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.9700000286102295, 'score_stable': 3.9700000286102295, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-09-13T11:32:17.683275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-13T11:32:18.790550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '56', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.6700000762939453, 'score_stable': 3.6700000762939453, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.309999942779541, 'raw_score_stable': 2.309999942779541, 'primary_affinity_weighted': 0.9772930145263672, 'average_primary_affinity': 0.9800000190734863, 'average_primary_affinity_weighted': 0.9800000190734863}}], 'osds': [{'osd': 0, 'uuid': '5a4d0d43-6562-441e-925e-4c992ee7b335', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6805', 'nonce': 2677506544}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 2677506544}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 2677506544}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 2677506544}]}, 'public_addr': '172.21.15.38:6805/2677506544', 'cluster_addr': '172.21.15.38:6806/2677506544', 'heartbeat_back_addr': '172.21.15.38:6808/2677506544', 'heartbeat_front_addr': '172.21.15.38:6807/2677506544', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '80665caa-6933-4f89-b454-128057e9ad92', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 3561806747}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6809', 'nonce': 3561806747}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6811', 'nonce': 3561806747}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 3561806747}]}, 'public_addr': '172.21.15.118:6808/3561806747', 'cluster_addr': '172.21.15.118:6809/3561806747', 'heartbeat_back_addr': '172.21.15.118:6811/3561806747', 'heartbeat_front_addr': '172.21.15.118:6810/3561806747', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'eb99fd51-a064-4311-932e-8b6d555409af', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 0, 'down_at': 41, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 4217282489}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 4217282489}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 4217282489}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 4217282489}]}, 'public_addr': '172.21.15.154:6800/4217282489', 'cluster_addr': '172.21.15.154:6801/4217282489', 'heartbeat_back_addr': '172.21.15.154:6803/4217282489', 'heartbeat_front_addr': '172.21.15.154:6802/4217282489', 'state': ['exists']}, {'osd': 3, 'uuid': '3d840ed9-a410-4bcb-b0f0-70fafdf69842', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.636688232421875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6804', 'nonce': 2326505959}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6805', 'nonce': 2326505959}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6807', 'nonce': 2326505959}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6806', 'nonce': 2326505959}]}, 'public_addr': '172.21.15.204:6804/2326505959', 'cluster_addr': '172.21.15.204:6805/2326505959', 'heartbeat_back_addr': '172.21.15.204:6807/2326505959', 'heartbeat_front_addr': '172.21.15.204:6806/2326505959', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '0c01b7d7-1309-40fa-913a-97d2a8499293', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6801', 'nonce': 3107658120}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 3107658120}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6804', 'nonce': 3107658120}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 3107658120}]}, 'public_addr': '172.21.15.38:6801/3107658120', 'cluster_addr': '172.21.15.38:6802/3107658120', 'heartbeat_back_addr': '172.21.15.38:6804/3107658120', 'heartbeat_front_addr': '172.21.15.38:6803/3107658120', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4cec928-1764-4fe0-8c5d-870d8a0c15c7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 3800782738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6801', 'nonce': 3800782738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6803', 'nonce': 3800782738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 3800782738}]}, 'public_addr': '172.21.15.118:6800/3800782738', 'cluster_addr': '172.21.15.118:6801/3800782738', 'heartbeat_back_addr': '172.21.15.118:6803/3800782738', 'heartbeat_front_addr': '172.21.15.118:6802/3800782738', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '89f6d810-2afc-42b0-b989-052218b6887c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 3986362716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 3986362716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 3986362716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 3986362716}]}, 'public_addr': '172.21.15.154:6808/3986362716', 'cluster_addr': '172.21.15.154:6809/3986362716', 'heartbeat_back_addr': '172.21.15.154:6811/3986362716', 'heartbeat_front_addr': '172.21.15.154:6810/3986362716', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ccfee49f-f394-4de0-8e30-853f38e31232', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6808', 'nonce': 1978982202}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6809', 'nonce': 1978982202}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6811', 'nonce': 1978982202}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6810', 'nonce': 1978982202}]}, 'public_addr': '172.21.15.204:6808/1978982202', 'cluster_addr': '172.21.15.204:6809/1978982202', 'heartbeat_back_addr': '172.21.15.204:6811/1978982202', 'heartbeat_front_addr': '172.21.15.204:6810/1978982202', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '03b1c505-1b92-4daa-9da4-c254be2ba792', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 41, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 3608171775}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 3608171775}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6816', 'nonce': 3608171775}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6815', 'nonce': 3608171775}]}, 'public_addr': '172.21.15.38:6813/3608171775', 'cluster_addr': '172.21.15.38:6814/3608171775', 'heartbeat_back_addr': '172.21.15.38:6816/3608171775', 'heartbeat_front_addr': '172.21.15.38:6815/3608171775', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '8fbee7a0-2ee6-4551-bf06-4e013e413fb4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 2580659844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6805', 'nonce': 2580659844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6807', 'nonce': 2580659844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 2580659844}]}, 'public_addr': '172.21.15.118:6804/2580659844', 'cluster_addr': '172.21.15.118:6805/2580659844', 'heartbeat_back_addr': '172.21.15.118:6807/2580659844', 'heartbeat_front_addr': '172.21.15.118:6806/2580659844', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '0808cb41-f62e-49b0-935d-616bea1a4db5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 2352891695}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 2352891695}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 2352891695}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 2352891695}]}, 'public_addr': '172.21.15.154:6812/2352891695', 'cluster_addr': '172.21.15.154:6813/2352891695', 'heartbeat_back_addr': '172.21.15.154:6815/2352891695', 'heartbeat_front_addr': '172.21.15.154:6814/2352891695', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '427dd574-f2a8-49e5-a32a-cfda023a514a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6800', 'nonce': 2304836889}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6801', 'nonce': 2304836889}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6803', 'nonce': 2304836889}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6802', 'nonce': 2304836889}]}, 'public_addr': '172.21.15.204:6800/2304836889', 'cluster_addr': '172.21.15.204:6801/2304836889', 'heartbeat_back_addr': '172.21.15.204:6803/2304836889', 'heartbeat_front_addr': '172.21.15.204:6802/2304836889', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '17942689-1130-4b6b-9f98-1605a03731b1', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 58, 'down_at': 59, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6809', 'nonce': 1688419254}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6810', 'nonce': 1688419254}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 1688419254}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6811', 'nonce': 1688419254}]}, 'public_addr': '172.21.15.38:6809/1688419254', 'cluster_addr': '172.21.15.38:6810/1688419254', 'heartbeat_back_addr': '172.21.15.38:6812/1688419254', 'heartbeat_front_addr': '172.21.15.38:6811/1688419254', 'state': ['exists']}, {'osd': 13, 'uuid': '62161cab-6d14-4571-971d-217601e38fbc', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 3727425991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6813', 'nonce': 3727425991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6815', 'nonce': 3727425991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 3727425991}]}, 'public_addr': '172.21.15.118:6812/3727425991', 'cluster_addr': '172.21.15.118:6813/3727425991', 'heartbeat_back_addr': '172.21.15.118:6815/3727425991', 'heartbeat_front_addr': '172.21.15.118:6814/3727425991', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '559ef141-27db-452d-9577-a6135421f64f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 538351706}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 538351706}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 538351706}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 538351706}]}, 'public_addr': '172.21.15.154:6804/538351706', 'cluster_addr': '172.21.15.154:6805/538351706', 'heartbeat_back_addr': '172.21.15.154:6807/538351706', 'heartbeat_front_addr': '172.21.15.154:6806/538351706', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '98f9ceba-79df-410a-9788-393057e79c30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 72, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6812', 'nonce': 501019151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6813', 'nonce': 501019151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6815', 'nonce': 501019151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.204:6814', 'nonce': 501019151}]}, 'public_addr': '172.21.15.204:6812/501019151', 'cluster_addr': '172.21.15.204:6813/501019151', 'heartbeat_back_addr': '172.21.15.204:6815/501019151', 'heartbeat_front_addr': '172.21.15.204:6814/501019151', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:03.855267+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:33:26.046732+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:32:00.198442+0000', 'dead_epoch': 41}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:04.310901+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:03.659118+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:33:09.623455+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.808086+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.499908+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:07.828658+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:33:22.980747+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:27.377283+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:26.397265+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:32:01.027609+0000', 'dead_epoch': 59}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.963999+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:19.534738+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-13T11:33:02.472566+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.3', 'osds': [7, 15, 13]}, {'pgid': '4.2', 'osds': [3, 12, 9]}, {'pgid': '4.4', 'osds': [3, 8, 5]}, {'pgid': '4.11', 'osds': [7, 3, 15]}, {'pgid': '4.12', 'osds': [14, 8, 3]}, {'pgid': '4.13', 'osds': [7, 14, 6]}, {'pgid': '4.14', 'osds': [4, 7, 3]}, {'pgid': '4.16', 'osds': [10, 6, 12]}, {'pgid': '4.19', 'osds': [6, 0, 7]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-13T11:33:37.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:33:37.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:37.102 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:37.096+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:37.136 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:37.199 INFO:tasks.rados.rados.0.smithi204.stdout:1720: finishing write tid 1 to smithi20438004-20 2024-09-13T11:33:37.199 INFO:tasks.rados.rados.0.smithi204.stdout:1720: finishing write tid 2 to smithi20438004-20 2024-09-13T11:33:37.199 INFO:tasks.rados.rados.0.smithi204.stdout:1721: expect (ObjNum 341 snap 0 seq_num 341) 2024-09-13T11:33:37.199 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:37.197+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:37.208 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1062642 2024-09-13T11:33:37.243 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:37.243 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:37.252 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:37.299 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:37.298+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:37.318 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1984391 2024-09-13T11:33:37.394 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:37.394 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:37.400 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:37.398+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:37.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:37.408 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2474124 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:1720: finishing write tid 3 to smithi20438004-20 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:1720: finishing write tid 4 to smithi20438004-20 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:1720: oid 20 updating version 0 to 101 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:1720: oid 20 version 101 is already newer than 99 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:1720: oid 20 version 101 is already newer than 100 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 101 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:1720: left oid 20 (ObjNum 359 snap 0 seq_num 359) 2024-09-13T11:33:37.434 INFO:tasks.rados.rados.0.smithi204.stdout:1717: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:37.435 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:37.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:33:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:33:37.439 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:33:37.451 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-13T11:33:36.946279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231016514,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":60,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":3919,"num_read_kb":2453252,"num_write":2662,"num_write_kb":622019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":112279409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":98,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":876736,"kb_used_data":531648,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1131585344,"statfs":{"total":1159641169920,"available":1158743392256,"internally_reserved":0,"allocated":544407552,"data_stored":539295448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19137,"internal_metadata":353154367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26346390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":347,"num_read_kb":296267,"num_write":176,"num_write_kb":51731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.601315"},"pg_stats":[{"pgid":"4.19","version":"53'20","reported_seq":191,"reported_epoch":80,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:29.345404+0000","last_change":"2024-09-13T11:33:12.028705+0000","last_active":"2024-09-13T11:33:29.345404+0000","last_peered":"2024-09-13T11:33:29.345404+0000","last_clean":"2024-09-13T11:33:29.345404+0000","last_became_active":"2024-09-13T11:32:59.281878+0000","last_became_peered":"2024-09-13T11:32:59.281878+0000","last_unstale":"2024-09-13T11:33:29.345404+0000","last_undegraded":"2024-09-13T11:33:29.345404+0000","last_fullsized":"2024-09-13T11:33:29.345404+0000","mapping_epoch":61,"log_start":"51'17","ondisk_log_start":"51'17","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'20","last_scrub_stamp":"2024-09-13T11:33:12.028588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.028588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:17.408780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6,0,7],"avail_no_missing":["6","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"80'83","reported_seq":309,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:31.054909+0000","last_change":"2024-09-13T11:33:16.278769+0000","last_active":"2024-09-13T11:33:31.054909+0000","last_peered":"2024-09-13T11:33:31.054909+0000","last_clean":"2024-09-13T11:33:31.054909+0000","last_became_active":"2024-09-13T11:33:16.278471+0000","last_became_peered":"2024-09-13T11:33:16.278471+0000","last_unstale":"2024-09-13T11:33:31.054909+0000","last_undegraded":"2024-09-13T11:33:31.054909+0000","last_fullsized":"2024-09-13T11:33:31.054909+0000","mapping_epoch":72,"log_start":"80'80","ondisk_log_start":"80'80","created":56,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'57","last_scrub_stamp":"2024-09-13T11:33:04.555283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.555283+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":78,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:32.651867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176655,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":44800,"num_write":32,"num_write_kb":7957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,6],"acting":[11,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.762624+0000","last_change":"2024-09-13T11:33:03.196131+0000","last_active":"2024-09-13T11:33:29.762624+0000","last_peered":"2024-09-13T11:33:29.762624+0000","last_clean":"2024-09-13T11:33:29.762624+0000","last_became_active":"2024-09-13T11:32:56.278133+0000","last_became_peered":"2024-09-13T11:32:56.278133+0000","last_unstale":"2024-09-13T11:33:29.762624+0000","last_undegraded":"2024-09-13T11:33:29.762624+0000","last_fullsized":"2024-09-13T11:33:29.762624+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.196058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:32.709794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6],"acting":[14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.16","version":"80'91","reported_seq":334,"reported_epoch":80,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:30.630073+0000","last_change":"2024-09-13T11:32:56.273159+0000","last_active":"2024-09-13T11:33:30.630073+0000","last_peered":"2024-09-13T11:33:30.630073+0000","last_clean":"2024-09-13T11:32:55.237210+0000","last_became_active":"2024-09-13T11:32:56.273159+0000","last_became_peered":"2024-09-13T11:32:56.273159+0000","last_unstale":"2024-09-13T11:33:30.630073+0000","last_undegraded":"2024-09-13T11:32:56.268297+0000","last_fullsized":"2024-09-13T11:32:56.268083+0000","mapping_epoch":58,"log_start":"74'88","ondisk_log_start":"74'88","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":87,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:32.113338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2092488,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":185,"num_read_kb":177096,"num_write":67,"num_write_kb":20499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":1}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"60'41","reported_seq":237,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.933516+0000","last_change":"2024-09-13T11:33:05.991060+0000","last_active":"2024-09-13T11:33:29.933516+0000","last_peered":"2024-09-13T11:33:29.933516+0000","last_clean":"2024-09-13T11:33:29.933516+0000","last_became_active":"2024-09-13T11:32:59.285266+0000","last_became_peered":"2024-09-13T11:32:59.285266+0000","last_unstale":"2024-09-13T11:33:29.933516+0000","last_undegraded":"2024-09-13T11:33:29.933516+0000","last_fullsized":"2024-09-13T11:33:29.933516+0000","mapping_epoch":61,"log_start":"52'36","ondisk_log_start":"52'36","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'41","last_scrub_stamp":"2024-09-13T11:33:05.990981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.990981+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":36,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:30.628856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,6],"acting":[0,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"80'105","reported_seq":394,"reported_epoch":80,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:31.867373+0000","last_change":"2024-09-13T11:33:04.630617+0000","last_active":"2024-09-13T11:33:31.867373+0000","last_peered":"2024-09-13T11:33:31.867373+0000","last_clean":"2024-09-13T11:33:31.867373+0000","last_became_active":"2024-09-13T11:32:58.287813+0000","last_became_peered":"2024-09-13T11:32:58.287813+0000","last_unstale":"2024-09-13T11:33:31.867373+0000","last_undegraded":"2024-09-13T11:33:31.867373+0000","last_fullsized":"2024-09-13T11:33:31.867373+0000","mapping_epoch":60,"log_start":"76'101","ondisk_log_start":"76'101","created":56,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'78","last_scrub_stamp":"2024-09-13T11:33:04.630512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.630512+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":101,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:28.662873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9647317,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":104901,"num_write":28,"num_write_kb":7807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,3],"avail_no_missing":["4","3","7"],"object_location_counts":[{"shards":"3,4,7","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"53'41","reported_seq":220,"reported_epoch":81,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:32.543855+0000","last_change":"2024-09-13T11:33:05.531596+0000","last_active":"2024-09-13T11:33:32.543855+0000","last_peered":"2024-09-13T11:33:32.543855+0000","last_clean":"2024-09-13T11:33:32.543855+0000","last_became_active":"2024-09-13T11:32:59.285096+0000","last_became_peered":"2024-09-13T11:32:59.285096+0000","last_unstale":"2024-09-13T11:33:32.543855+0000","last_undegraded":"2024-09-13T11:33:32.543855+0000","last_fullsized":"2024-09-13T11:33:32.543855+0000","mapping_epoch":61,"log_start":"50'38","ondisk_log_start":"50'38","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'41","last_scrub_stamp":"2024-09-13T11:33:05.531507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.531507+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:13.447752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,14],"acting":[7,14,6],"avail_no_missing":["7","6","14"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"80'126","reported_seq":417,"reported_epoch":80,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:31.830084+0000","last_change":"2024-09-13T11:33:05.196910+0000","last_active":"2024-09-13T11:33:31.830084+0000","last_peered":"2024-09-13T11:33:31.830084+0000","last_clean":"2024-09-13T11:33:31.830084+0000","last_became_active":"2024-09-13T11:32:59.288348+0000","last_became_peered":"2024-09-13T11:32:59.288348+0000","last_unstale":"2024-09-13T11:33:31.830084+0000","last_undegraded":"2024-09-13T11:33:31.830084+0000","last_fullsized":"2024-09-13T11:33:31.830084+0000","mapping_epoch":61,"log_start":"80'123","ondisk_log_start":"80'123","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'93","last_scrub_stamp":"2024-09-13T11:33:05.196778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.196778+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":123,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:09.913125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10739772,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":92,"num_read_kb":87547,"num_write":40,"num_write_kb":12274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8],"acting":[14,8,3],"avail_no_missing":["14","3","8"],"object_location_counts":[{"shards":"3,8,14","objects":3}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"81'140","reported_seq":481,"reported_epoch":81,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:32.698208+0000","last_change":"2024-09-13T11:33:03.598922+0000","last_active":"2024-09-13T11:33:32.698208+0000","last_peered":"2024-09-13T11:33:32.698208+0000","last_clean":"2024-09-13T11:33:32.698208+0000","last_became_active":"2024-09-13T11:32:57.287130+0000","last_became_peered":"2024-09-13T11:32:57.287130+0000","last_unstale":"2024-09-13T11:33:32.698208+0000","last_undegraded":"2024-09-13T11:33:32.698208+0000","last_fullsized":"2024-09-13T11:33:32.698208+0000","mapping_epoch":59,"log_start":"81'137","ondisk_log_start":"81'137","created":56,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'103","last_scrub_stamp":"2024-09-13T11:33:03.598827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.598827+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":136,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:29.147515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10538088,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":172551,"num_write":37,"num_write_kb":14525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,15],"avail_no_missing":["7","3","15"],"object_location_counts":[{"shards":"3,7,15","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"76'68","reported_seq":304,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.937557+0000","last_change":"2024-09-13T11:33:12.208901+0000","last_active":"2024-09-13T11:33:29.937557+0000","last_peered":"2024-09-13T11:33:29.937557+0000","last_clean":"2024-09-13T11:33:29.937557+0000","last_became_active":"2024-09-13T11:32:59.280441+0000","last_became_peered":"2024-09-13T11:32:59.280441+0000","last_unstale":"2024-09-13T11:33:29.937557+0000","last_undegraded":"2024-09-13T11:33:29.937557+0000","last_fullsized":"2024-09-13T11:33:29.937557+0000","mapping_epoch":61,"log_start":"76'65","ondisk_log_start":"76'65","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'60","last_scrub_stamp":"2024-09-13T11:33:12.208834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.208834+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:21.081399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":26650,"num_write":10,"num_write_kb":2746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,3],"acting":[10,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":133,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T11:33:32.584367+0000","last_change":"2024-09-13T11:32:45.278444+0000","last_active":"2024-09-13T11:33:32.584367+0000","last_peered":"2024-09-13T11:33:32.584367+0000","last_clean":"2024-09-13T11:33:32.584367+0000","last_became_active":"2024-09-13T11:32:45.277643+0000","last_became_peered":"2024-09-13T11:32:45.277643+0000","last_unstale":"2024-09-13T11:33:32.584367+0000","last_undegraded":"2024-09-13T11:33:32.584367+0000","last_fullsized":"2024-09-13T11:33:32.584367+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:36.144116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,7],"acting":[15,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.1","version":"79'122","reported_seq":424,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T11:33:32.666514+0000","last_change":"2024-09-13T11:33:01.310446+0000","last_active":"2024-09-13T11:33:32.666514+0000","last_peered":"2024-09-13T11:33:32.666514+0000","last_clean":"2024-09-13T11:33:32.666514+0000","last_became_active":"2024-09-13T11:32:54.604852+0000","last_became_peered":"2024-09-13T11:32:54.604852+0000","last_unstale":"2024-09-13T11:33:32.666514+0000","last_undegraded":"2024-09-13T11:33:32.666514+0000","last_fullsized":"2024-09-13T11:33:32.666514+0000","mapping_epoch":52,"log_start":"75'119","ondisk_log_start":"75'119","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'105","last_scrub_stamp":"2024-09-13T11:33:01.310380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.310380+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":117,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:08.141302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5277464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":52775,"num_write":16,"num_write_kb":6395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3,7],"acting":[15,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.762697+0000","last_change":"2024-09-13T11:32:08.625099+0000","last_active":"2024-09-13T11:33:29.762697+0000","last_peered":"2024-09-13T11:33:29.762697+0000","last_clean":"2024-09-13T11:33:29.762697+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:33:29.762697+0000","last_undegraded":"2024-09-13T11:33:29.762697+0000","last_fullsized":"2024-09-13T11:33:29.762697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:31.093378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"80'83","reported_seq":322,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:31.076839+0000","last_change":"2024-09-13T11:33:02.247329+0000","last_active":"2024-09-13T11:33:31.076839+0000","last_peered":"2024-09-13T11:33:31.076839+0000","last_clean":"2024-09-13T11:33:31.076839+0000","last_became_active":"2024-09-13T11:32:54.604560+0000","last_became_peered":"2024-09-13T11:32:54.604560+0000","last_unstale":"2024-09-13T11:33:31.076839+0000","last_undegraded":"2024-09-13T11:33:31.076839+0000","last_fullsized":"2024-09-13T11:33:31.076839+0000","mapping_epoch":51,"log_start":"80'80","ondisk_log_start":"80'80","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'57","last_scrub_stamp":"2024-09-13T11:33:02.247255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.247255+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:19.872698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6605217,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":82857,"num_write":32,"num_write_kb":9755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":129,"reported_epoch":80,"state":"active+undersized","last_fresh":"2024-09-13T11:33:30.210846+0000","last_change":"2024-09-13T11:32:32.498855+0000","last_active":"2024-09-13T11:33:30.210846+0000","last_peered":"2024-09-13T11:33:30.210846+0000","last_clean":"2024-09-13T11:32:30.479480+0000","last_became_active":"2024-09-13T11:32:32.498855+0000","last_became_peered":"2024-09-13T11:32:32.498855+0000","last_unstale":"2024-09-13T11:33:30.210846+0000","last_undegraded":"2024-09-13T11:33:30.210846+0000","last_fullsized":"2024-09-13T11:32:32.497672+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:35.325284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":["8","6"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"80'55","reported_seq":226,"reported_epoch":80,"state":"active+undersized","last_fresh":"2024-09-13T11:33:30.785008+0000","last_change":"2024-09-13T11:33:26.485108+0000","last_active":"2024-09-13T11:33:30.785008+0000","last_peered":"2024-09-13T11:33:30.785008+0000","last_clean":"2024-09-13T11:32:49.952102+0000","last_became_active":"2024-09-13T11:32:57.283497+0000","last_became_peered":"2024-09-13T11:32:57.283497+0000","last_unstale":"2024-09-13T11:33:30.785008+0000","last_undegraded":"2024-09-13T11:33:30.785008+0000","last_fullsized":"2024-09-13T11:32:57.277969+0000","mapping_epoch":59,"log_start":"80'52","ondisk_log_start":"80'52","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":52,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:32.050551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462209,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":37531,"num_write":36,"num_write_kb":11506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2383182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":["11","6"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":128,"reported_epoch":80,"state":"active+undersized","last_fresh":"2024-09-13T11:33:29.345264+0000","last_change":"2024-09-13T11:32:57.283324+0000","last_active":"2024-09-13T11:33:29.345264+0000","last_peered":"2024-09-13T11:33:29.345264+0000","last_clean":"2024-09-13T11:32:55.276320+0000","last_became_active":"2024-09-13T11:32:57.283324+0000","last_became_peered":"2024-09-13T11:32:57.283324+0000","last_unstale":"2024-09-13T11:33:29.345264+0000","last_undegraded":"2024-09-13T11:33:29.345264+0000","last_fullsized":"2024-09-13T11:32:57.273798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:29.914763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"80'138","reported_seq":400,"reported_epoch":80,"state":"active+undersized+remapped","last_fresh":"2024-09-13T11:33:31.078989+0000","last_change":"2024-09-13T11:32:57.284225+0000","last_active":"2024-09-13T11:33:31.078989+0000","last_peered":"2024-09-13T11:33:31.078989+0000","last_clean":"2024-09-13T11:32:55.273515+0000","last_became_active":"2024-09-13T11:32:57.284225+0000","last_became_peered":"2024-09-13T11:32:57.284225+0000","last_unstale":"2024-09-13T11:33:31.078989+0000","last_undegraded":"2024-09-13T11:33:31.078989+0000","last_fullsized":"2024-09-13T11:32:57.278637+0000","mapping_epoch":59,"log_start":"80'135","ondisk_log_start":"80'135","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":132,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:34.842960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5949760,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":175,"num_read_kb":145402,"num_write":106,"num_write_kb":31642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4501279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3,9],"avail_no_missing":["3","9"],"object_location_counts":[{"shards":"3,9","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"80'278","reported_seq":477,"reported_epoch":81,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:32.544082+0000","last_change":"2024-09-13T11:33:29.491664+0000","last_active":"2024-09-13T11:33:32.544082+0000","last_peered":"2024-09-13T11:33:32.544082+0000","last_clean":"2024-09-13T11:33:32.544082+0000","last_became_active":"2024-09-13T11:32:48.281772+0000","last_became_peered":"2024-09-13T11:32:48.281772+0000","last_unstale":"2024-09-13T11:33:32.544082+0000","last_undegraded":"2024-09-13T11:33:32.544082+0000","last_fullsized":"2024-09-13T11:33:32.544082+0000","mapping_epoch":52,"log_start":"80'275","ondisk_log_start":"80'275","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'252","last_scrub_stamp":"2024-09-13T11:33:29.491435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:33:29.491435+0000","objects_scrubbed":42,"log_size":3,"log_dups_size":284,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:49.437842+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27752918,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":46,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":250,"num_read_kb":9003,"num_write":365,"num_write_kb":73371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":14112815,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15],"acting":[7,15,13],"avail_no_missing":["7","13","15"],"object_location_counts":[{"shards":"7,13,15","objects":46}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":145,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.920596+0000","last_change":"2024-09-13T11:33:26.770230+0000","last_active":"2024-09-13T11:33:29.920596+0000","last_peered":"2024-09-13T11:33:29.920596+0000","last_clean":"2024-09-13T11:33:29.920596+0000","last_became_active":"2024-09-13T11:32:43.554077+0000","last_became_peered":"2024-09-13T11:32:43.554077+0000","last_unstale":"2024-09-13T11:33:29.920596+0000","last_undegraded":"2024-09-13T11:33:29.920596+0000","last_fullsized":"2024-09-13T11:33:29.920596+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:33:26.770137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:26.770137+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:51.278083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,11],"acting":[3,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"75'81","reported_seq":324,"reported_epoch":80,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:29.920451+0000","last_change":"2024-09-13T11:33:01.558907+0000","last_active":"2024-09-13T11:33:29.920451+0000","last_peered":"2024-09-13T11:33:29.920451+0000","last_clean":"2024-09-13T11:33:29.920451+0000","last_became_active":"2024-09-13T11:32:54.600364+0000","last_became_peered":"2024-09-13T11:32:54.600364+0000","last_unstale":"2024-09-13T11:33:29.920451+0000","last_undegraded":"2024-09-13T11:33:29.920451+0000","last_fullsized":"2024-09-13T11:33:29.920451+0000","mapping_epoch":49,"log_start":"74'78","ondisk_log_start":"74'78","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'73","last_scrub_stamp":"2024-09-13T11:33:01.558809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.558809+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:23.044953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857223,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":45061,"num_write":9,"num_write_kb":2663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8,5],"avail_no_missing":["3","5","8"],"object_location_counts":[{"shards":"3,5,8","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"81'271","reported_seq":470,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:32.663913+0000","last_change":"2024-09-13T11:32:57.280350+0000","last_active":"2024-09-13T11:33:32.663913+0000","last_peered":"2024-09-13T11:33:32.663913+0000","last_clean":"2024-09-13T11:32:54.594742+0000","last_became_active":"2024-09-13T11:32:57.280350+0000","last_became_peered":"2024-09-13T11:32:57.280350+0000","last_unstale":"2024-09-13T11:33:32.663913+0000","last_undegraded":"2024-09-13T11:32:57.275975+0000","last_fullsized":"2024-09-13T11:32:57.275775+0000","mapping_epoch":59,"log_start":"80'268","ondisk_log_start":"80'268","created":30,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":274,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:32.824670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27678815,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":248,"num_read_kb":12152,"num_write":350,"num_write_kb":76754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":24388633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6],"acting":[15,6],"avail_no_missing":["15","6"],"object_location_counts":[{"shards":"6,15","objects":35}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.762584+0000","last_change":"2024-09-13T11:33:28.020586+0000","last_active":"2024-09-13T11:33:29.762584+0000","last_peered":"2024-09-13T11:33:29.762584+0000","last_clean":"2024-09-13T11:33:29.762584+0000","last_became_active":"2024-09-13T11:32:45.284134+0000","last_became_peered":"2024-09-13T11:32:45.284134+0000","last_unstale":"2024-09-13T11:33:29.762584+0000","last_undegraded":"2024-09-13T11:33:29.762584+0000","last_fullsized":"2024-09-13T11:33:29.762584+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:28.020512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:28.020512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:48.300493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,8],"acting":[14,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":141,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.935428+0000","last_change":"2024-09-13T11:33:03.236076+0000","last_active":"2024-09-13T11:33:29.935428+0000","last_peered":"2024-09-13T11:33:29.935428+0000","last_clean":"2024-09-13T11:33:29.935428+0000","last_became_active":"2024-09-13T11:32:54.601517+0000","last_became_peered":"2024-09-13T11:32:54.601517+0000","last_unstale":"2024-09-13T11:33:29.935428+0000","last_undegraded":"2024-09-13T11:33:29.935428+0000","last_fullsized":"2024-09-13T11:33:29.935428+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.236001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.236001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:06.939201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"80'306","reported_seq":521,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:31.845801+0000","last_change":"2024-09-13T11:32:47.277934+0000","last_active":"2024-09-13T11:33:31.845801+0000","last_peered":"2024-09-13T11:33:31.845801+0000","last_clean":"2024-09-13T11:33:31.845801+0000","last_became_active":"2024-09-13T11:32:47.277392+0000","last_became_peered":"2024-09-13T11:32:47.277392+0000","last_unstale":"2024-09-13T11:33:31.845801+0000","last_undegraded":"2024-09-13T11:33:31.845801+0000","last_fullsized":"2024-09-13T11:33:31.845801+0000","mapping_epoch":51,"log_start":"80'303","ondisk_log_start":"80'303","created":30,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":318,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:40.554823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26178407,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":266,"num_read_kb":6217,"num_write":378,"num_write_kb":85360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":28975184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,0],"acting":[6,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":124,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.920368+0000","last_change":"2024-09-13T11:32:08.377316+0000","last_active":"2024-09-13T11:33:29.920368+0000","last_peered":"2024-09-13T11:33:29.920368+0000","last_clean":"2024-09-13T11:33:29.920368+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:33:29.920368+0000","last_undegraded":"2024-09-13T11:33:29.920368+0000","last_fullsized":"2024-09-13T11:33:29.920368+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"80'83","reported_seq":333,"reported_epoch":80,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:31.872445+0000","last_change":"2024-09-13T11:32:57.280636+0000","last_active":"2024-09-13T11:33:31.872445+0000","last_peered":"2024-09-13T11:33:31.872445+0000","last_clean":"2024-09-13T11:32:54.904504+0000","last_became_active":"2024-09-13T11:32:57.280636+0000","last_became_peered":"2024-09-13T11:32:57.280636+0000","last_unstale":"2024-09-13T11:33:31.872445+0000","last_undegraded":"2024-09-13T11:32:57.277736+0000","last_fullsized":"2024-09-13T11:32:57.277594+0000","mapping_epoch":59,"log_start":"74'80","ondisk_log_start":"74'80","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":78,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:33.088593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":196,"num_read_kb":151219,"num_write":51,"num_write_kb":17057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"81'336","reported_seq":528,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:32.584242+0000","last_change":"2024-09-13T11:33:16.277928+0000","last_active":"2024-09-13T11:33:32.584242+0000","last_peered":"2024-09-13T11:33:32.584242+0000","last_clean":"2024-09-13T11:32:55.257840+0000","last_became_active":"2024-09-13T11:33:16.277928+0000","last_became_peered":"2024-09-13T11:33:16.277928+0000","last_unstale":"2024-09-13T11:33:32.584242+0000","last_undegraded":"2024-09-13T11:33:16.272371+0000","last_fullsized":"2024-09-13T11:33:16.272253+0000","mapping_epoch":72,"log_start":"80'333","ondisk_log_start":"80'333","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":348,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:35.602877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15736858,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":306,"num_read_kb":5435,"num_write":429,"num_write_kb":85637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":16251716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3],"acting":[15,3],"avail_no_missing":["15","3"],"object_location_counts":[{"shards":"3,15","objects":29}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":110,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T11:33:32.584261+0000","last_change":"2024-09-13T11:33:27.080910+0000","last_active":"2024-09-13T11:33:32.584261+0000","last_peered":"2024-09-13T11:33:32.584261+0000","last_clean":"2024-09-13T11:33:32.584261+0000","last_became_active":"2024-09-13T11:32:45.284150+0000","last_became_peered":"2024-09-13T11:32:45.284150+0000","last_unstale":"2024-09-13T11:33:32.584261+0000","last_undegraded":"2024-09-13T11:33:32.584261+0000","last_fullsized":"2024-09-13T11:33:32.584261+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:33:27.080847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:27.080847+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:31.383253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,0],"acting":[15,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":243,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.933536+0000","last_change":"2024-09-13T11:33:16.944043+0000","last_active":"2024-09-13T11:33:29.933536+0000","last_peered":"2024-09-13T11:33:29.933536+0000","last_clean":"2024-09-13T11:33:29.933536+0000","last_became_active":"2024-09-13T11:32:48.286550+0000","last_became_peered":"2024-09-13T11:32:48.286550+0000","last_unstale":"2024-09-13T11:33:29.933536+0000","last_undegraded":"2024-09-13T11:33:29.933536+0000","last_fullsized":"2024-09-13T11:33:29.933536+0000","mapping_epoch":52,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:33:16.943930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:33:16.943930+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:23.725756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"76'46","reported_seq":243,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.935932+0000","last_change":"2024-09-13T11:33:02.151042+0000","last_active":"2024-09-13T11:33:29.935932+0000","last_peered":"2024-09-13T11:33:29.935932+0000","last_clean":"2024-09-13T11:33:29.935932+0000","last_became_active":"2024-09-13T11:32:54.609316+0000","last_became_peered":"2024-09-13T11:32:54.609316+0000","last_unstale":"2024-09-13T11:33:29.935932+0000","last_undegraded":"2024-09-13T11:33:29.935932+0000","last_fullsized":"2024-09-13T11:33:29.935932+0000","mapping_epoch":50,"log_start":"76'43","ondisk_log_start":"76'43","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'41","last_scrub_stamp":"2024-09-13T11:33:02.150975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.150975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":42,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:24.325318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311012,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"80'54","reported_seq":263,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:31.830187+0000","last_change":"2024-09-13T11:33:01.646865+0000","last_active":"2024-09-13T11:33:31.830187+0000","last_peered":"2024-09-13T11:33:31.830187+0000","last_clean":"2024-09-13T11:33:31.830187+0000","last_became_active":"2024-09-13T11:32:54.601608+0000","last_became_peered":"2024-09-13T11:32:54.601608+0000","last_unstale":"2024-09-13T11:33:31.830187+0000","last_undegraded":"2024-09-13T11:33:31.830187+0000","last_fullsized":"2024-09-13T11:33:31.830187+0000","mapping_epoch":31,"log_start":"61'51","ondisk_log_start":"61'51","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'53","last_scrub_stamp":"2024-09-13T11:33:01.646779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.646779+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:10.069513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546277,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":71001,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"78'53","reported_seq":254,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:30.581305+0000","last_change":"2024-09-13T11:33:02.042709+0000","last_active":"2024-09-13T11:33:30.581305+0000","last_peered":"2024-09-13T11:33:30.581305+0000","last_clean":"2024-09-13T11:33:30.581305+0000","last_became_active":"2024-09-13T11:32:54.604804+0000","last_became_peered":"2024-09-13T11:32:54.604804+0000","last_unstale":"2024-09-13T11:33:30.581305+0000","last_undegraded":"2024-09-13T11:33:30.581305+0000","last_fullsized":"2024-09-13T11:33:30.581305+0000","mapping_epoch":31,"log_start":"78'50","ondisk_log_start":"78'50","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2024-09-13T11:33:02.042611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.042611+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:02.206008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5966551,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":64691,"num_write":23,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14,7],"acting":[0,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"77'70","reported_seq":268,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.954597+0000","last_change":"2024-09-13T11:32:47.284684+0000","last_active":"2024-09-13T11:33:29.954597+0000","last_peered":"2024-09-13T11:33:29.954597+0000","last_clean":"2024-09-13T11:33:29.954597+0000","last_became_active":"2024-09-13T11:32:47.279408+0000","last_became_peered":"2024-09-13T11:32:47.279408+0000","last_unstale":"2024-09-13T11:33:29.954597+0000","last_undegraded":"2024-09-13T11:33:29.954597+0000","last_fullsized":"2024-09-13T11:33:29.954597+0000","mapping_epoch":51,"log_start":"75'67","ondisk_log_start":"75'67","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":67,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2941258,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":143,"num_read_kb":122924,"num_write":77,"num_write_kb":20518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2979858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"76'68","reported_seq":266,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:29.936074+0000","last_change":"2024-09-13T11:32:48.288341+0000","last_active":"2024-09-13T11:33:29.936074+0000","last_peered":"2024-09-13T11:33:29.936074+0000","last_clean":"2024-09-13T11:33:29.936074+0000","last_became_active":"2024-09-13T11:32:48.283697+0000","last_became_peered":"2024-09-13T11:32:48.283697+0000","last_unstale":"2024-09-13T11:33:29.936074+0000","last_undegraded":"2024-09-13T11:33:29.936074+0000","last_fullsized":"2024-09-13T11:33:29.936074+0000","mapping_epoch":52,"log_start":"76'65","ondisk_log_start":"76'65","created":31,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":65,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:46.169371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302691,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":167,"num_read_kb":147744,"num_write":70,"num_write_kb":20470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4770005,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"80'38","reported_seq":218,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:32.584196+0000","last_change":"2024-09-13T11:33:27.445927+0000","last_active":"2024-09-13T11:33:32.584196+0000","last_peered":"2024-09-13T11:33:32.584196+0000","last_clean":"2024-09-13T11:32:55.276844+0000","last_became_active":"2024-09-13T11:32:57.282855+0000","last_became_peered":"2024-09-13T11:32:57.282855+0000","last_unstale":"2024-09-13T11:33:32.584196+0000","last_undegraded":"2024-09-13T11:33:27.445134+0000","last_fullsized":"2024-09-13T11:32:57.277023+0000","mapping_epoch":59,"log_start":"78'35","ondisk_log_start":"78'35","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":35,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:37.113951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":58667,"num_write":35,"num_write_kb":11475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2040825,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7],"acting":[15,7],"avail_no_missing":["15","7"],"object_location_counts":[{"shards":"7,15","objects":1}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.d","version":"80'98","reported_seq":329,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T11:33:31.846976+0000","last_change":"2024-09-13T11:33:16.280221+0000","last_active":"2024-09-13T11:33:31.846976+0000","last_peered":"2024-09-13T11:33:31.846976+0000","last_clean":"2024-09-13T11:33:31.846976+0000","last_became_active":"2024-09-13T11:33:16.279717+0000","last_became_peered":"2024-09-13T11:33:16.279717+0000","last_unstale":"2024-09-13T11:33:31.846976+0000","last_undegraded":"2024-09-13T11:33:31.846976+0000","last_fullsized":"2024-09-13T11:33:31.846976+0000","mapping_epoch":72,"log_start":"80'95","ondisk_log_start":"80'95","created":31,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":95,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:48.989249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9848968,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":277,"num_read_kb":207535,"num_write":100,"num_write_kb":27943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"81'96","reported_seq":384,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T11:33:32.653364+0000","last_change":"2024-09-13T11:32:47.281013+0000","last_active":"2024-09-13T11:33:32.653364+0000","last_peered":"2024-09-13T11:33:32.653364+0000","last_clean":"2024-09-13T11:33:32.653364+0000","last_became_active":"2024-09-13T11:32:47.277862+0000","last_became_peered":"2024-09-13T11:32:47.277862+0000","last_unstale":"2024-09-13T11:33:32.653364+0000","last_undegraded":"2024-09-13T11:33:32.653364+0000","last_fullsized":"2024-09-13T11:33:32.653364+0000","mapping_epoch":51,"log_start":"81'93","ondisk_log_start":"81'93","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":93,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13160013,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":352,"num_read_kb":274929,"num_write":99,"num_write_kb":31525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5522961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,0],"acting":[15,14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.f","version":"80'117","reported_seq":404,"reported_epoch":80,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:31.868901+0000","last_change":"2024-09-13T11:32:57.287440+0000","last_active":"2024-09-13T11:33:31.868901+0000","last_peered":"2024-09-13T11:33:31.868901+0000","last_clean":"2024-09-13T11:32:55.848229+0000","last_became_active":"2024-09-13T11:32:57.287440+0000","last_became_peered":"2024-09-13T11:32:57.287440+0000","last_unstale":"2024-09-13T11:33:31.868901+0000","last_undegraded":"2024-09-13T11:32:57.280313+0000","last_fullsized":"2024-09-13T11:32:57.280204+0000","mapping_epoch":59,"log_start":"80'114","ondisk_log_start":"80'114","created":31,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:33.810129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12958329,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":347,"num_read_kb":344490,"num_write":120,"num_write_kb":33304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5762564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":["10","14"],"object_location_counts":[{"shards":"10,14","objects":4}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":133079129,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2763,"num_read_kb":2420372,"num_write":993,"num_write_kb":298223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":27960674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":323547136,"data_stored":323044089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":64,"acting":72,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":97346998,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":64,"num_objects_misplaced":46,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":1070,"num_read_kb":32807,"num_write":1522,"num_write_kb":321122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":83728348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":367607808,"data_stored":366741289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":9,"acting":10,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":22,"acting":22,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":124124,"kb_used_data":92940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94247716,"statfs":{"total":96636764160,"available":96509661184,"internally_reserved":0,"allocated":95170560,"data_stored":94665610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":98784247829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247829,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89668,"kb_used_data":58484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94282172,"statfs":{"total":96636764160,"available":96544944128,"internally_reserved":0,"allocated":59887616,"data_stored":59480502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280532,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81484,"kb_used_data":50300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94290356,"statfs":{"total":96636764160,"available":96553324544,"internally_reserved":0,"allocated":51507200,"data_stored":51023460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54680,"kb_used_data":27656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94317160,"statfs":{"total":96636764160,"available":96580771840,"internally_reserved":0,"allocated":28319744,"data_stored":27912548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":170144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95840,"kb_used_data":64656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94276000,"statfs":{"total":96636764160,"available":96538624000,"internally_reserved":0,"allocated":66207744,"data_stored":65718636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,4,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280532,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48420,"kb_used_data":21396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323420,"statfs":{"total":96636764160,"available":96587182080,"internally_reserved":0,"allocated":21909504,"data_stored":21501690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":94489280532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280533,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98544,"kb_used_data":67360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94273296,"statfs":{"total":96636764160,"available":96535855104,"internally_reserved":0,"allocated":68976640,"data_stored":68445391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44288,"kb_used_data":17264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327552,"statfs":{"total":96636764160,"available":96591413248,"internally_reserved":0,"allocated":17678336,"data_stored":17281106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280532,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49600,"kb_used_data":22576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322240,"statfs":{"total":96636764160,"available":96585973760,"internally_reserved":0,"allocated":23117824,"data_stored":22702485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313236,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67380,"kb_used_data":40356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94304460,"statfs":{"total":96636764160,"available":96567767040,"internally_reserved":0,"allocated":41324544,"data_stored":40838820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,6,7,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95324,"kb_used_data":68300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276516,"statfs":{"total":96636764160,"available":96539152384,"internally_reserved":0,"allocated":69939200,"data_stored":69555056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26251264,"data_stored":26178407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24444928,"data_stored":24413026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23801856,"data_stored":23781575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14548992,"data_stored":14506010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":54202368,"data_stored":54064070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":27836416,"data_stored":27752918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28405760,"data_stored":28348291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":26251264,"data_stored":26178407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":42668032,"data_stored":42597076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":27836416,"data_stored":27752918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":71360512,"data_stored":71168591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23752704,"data_stored":23716858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18067456,"data_stored":18030231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4225584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50151424,"data_stored":50084412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":21000192,"data_stored":20963863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2412544,"data_stored":2410759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13864960,"data_stored":13843494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":31252480,"data_stored":31207248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16195584,"data_stored":16173361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4681728,"data_stored":4675020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":22204416,"data_stored":22164639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14159872,"data_stored":14122567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":26669056,"data_stored":26630598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":24473600,"data_stored":24444051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":27410432,"data_stored":27374721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":23011328,"data_stored":22976683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:33:37.455 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [6, 0, 4, 7, 12, 3] 2024-09-13T11:33:37.455 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '6', '0', '4', '7', '12', '3'] 2024-09-13T11:33:37.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.7 6 0 4 7 12 3 2024-09-13T11:33:37.501 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:37.499+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:37.513 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:37.521 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:37.521 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:37.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:37.602 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:37.600+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: finishing write tid 1 to smithi20438004-27 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: finishing write tid 2 to smithi20438004-27 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: finishing write tid 3 to smithi20438004-27 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: finishing write tid 4 to smithi20438004-27 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: oid 27 updating version 0 to 120 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: oid 27 version 120 is already newer than 119 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: oid 27 updating version 120 to 121 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 121 (ObjNum 360 snap 0 seq_num 360) dirty exists 2024-09-13T11:33:37.673 INFO:tasks.rados.rados.0.smithi204.stdout:1724: left oid 27 (ObjNum 360 snap 0 seq_num 360) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1712: done (15 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1713: done (14 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1714: done (13 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1715: done (12 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1716: done (11 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1717: done (10 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1718: done (9 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1720: done (8 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1721: done (7 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1722: done (6 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1723: done (5 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1724: done (4 left) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1729: read oid 25 snap -1 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1729: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-13T11:33:37.674 INFO:tasks.rados.rados.0.smithi204.stdout:1730: read oid 12 snap -1 2024-09-13T11:33:37.675 INFO:tasks.rados.rados.0.smithi204.stdout:1730: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-13T11:33:37.675 INFO:tasks.rados.rados.0.smithi204.stdout:1731: write oid 4 current snap is 0 2024-09-13T11:33:37.675 INFO:tasks.rados.rados.0.smithi204.stdout:1726: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:37.675 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:37.675 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:37.675 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:37.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:37.727 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:37.726+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:37.752 INFO:tasks.rados.rados.0.smithi204.stdout:1725: finishing set_chunk to oid 47 2024-09-13T11:33:37.753 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 46 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-09-13T11:33:37.753 INFO:tasks.rados.rados.0.smithi204.stdout:1727: expect (ObjNum 336 snap 0 seq_num 336) 2024-09-13T11:33:37.767 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1938191 2024-09-13T11:33:37.788 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:37.788 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:37.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:37.803 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:37.802+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:37.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:37.904 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:37.903+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:37.911 INFO:tasks.rados.rados.0.smithi204.stdout:1728: finishing set_chunk to oid 5 2024-09-13T11:33:37.911 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 83 (ObjNum 343 snap 0 seq_num 343) dirty exists 2024-09-13T11:33:37.911 INFO:tasks.rados.rados.0.smithi204.stdout:1729: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-13T11:33:37.939 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:37.939 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:37.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:38.005 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:38.003+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:38.018 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:38.050 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:38.050 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:38.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:38.074 INFO:tasks.rados.rados.0.smithi204.stdout:1730: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-13T11:33:38.074 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:38.106 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:38.104+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:38.143 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:38.162 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:38.162 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:38.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:38.207 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:38.205+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:38.212 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:38.274 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:38.274 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:38.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:38.283 INFO:tasks.rados.rados.0.smithi204.stdout:1731: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-13T11:33:38.288 INFO:tasks.rados.rados.0.smithi204.stdout:1731: writing smithi20438004-4 from 467327 to 1100807 tid 1 2024-09-13T11:33:38.293 INFO:tasks.rados.rados.0.smithi204.stdout:1731: writing smithi20438004-4 from 1825176 to 2067327 tid 2 2024-09-13T11:33:38.293 INFO:tasks.rados.rados.0.smithi204.stdout:1725: done (6 left) 2024-09-13T11:33:38.294 INFO:tasks.rados.rados.0.smithi204.stdout:1726: done (5 left) 2024-09-13T11:33:38.294 INFO:tasks.rados.rados.0.smithi204.stdout:1727: done (4 left) 2024-09-13T11:33:38.294 INFO:tasks.rados.rados.0.smithi204.stdout:1728: done (3 left) 2024-09-13T11:33:38.294 INFO:tasks.rados.rados.0.smithi204.stdout:1729: done (2 left) 2024-09-13T11:33:38.294 INFO:tasks.rados.rados.0.smithi204.stdout:1730: done (1 left) 2024-09-13T11:33:38.294 INFO:tasks.rados.rados.0.smithi204.stdout:1732: delete oid 48 current snap is 0 2024-09-13T11:33:38.294 INFO:tasks.rados.rados.0.smithi204.stdout:1731: finishing write tid 1 to smithi20438004-4 2024-09-13T11:33:38.295 INFO:tasks.rados.rados.0.smithi204.stdout:1732: done (1 left) 2024-09-13T11:33:38.295 INFO:tasks.rados.rados.0.smithi204.stdout:1733: write oid 24 current snap is 0 2024-09-13T11:33:38.295 INFO:tasks.rados.rados.0.smithi204.stdout:1733: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-13T11:33:38.301 INFO:tasks.rados.rados.0.smithi204.stdout:1733: writing smithi20438004-24 from 500941 to 1284919 tid 1 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1733: writing smithi20438004-24 from 1942140 to 2100941 tid 2 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1734: read oid 1 snap -1 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1734: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1735: delete oid 14 current snap is 0 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1731: finishing write tid 2 to smithi20438004-4 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1731: finishing write tid 3 to smithi20438004-4 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1731: finishing write tid 4 to smithi20438004-4 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1731: oid 4 updating version 0 to 86 2024-09-13T11:33:38.307 INFO:tasks.rados.rados.0.smithi204.stdout:1731: oid 4 version 86 is already newer than 84 2024-09-13T11:33:38.308 INFO:tasks.rados.rados.0.smithi204.stdout:1731: oid 4 version 86 is already newer than 85 2024-09-13T11:33:38.308 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 86 (ObjNum 361 snap 0 seq_num 361) dirty exists 2024-09-13T11:33:38.308 INFO:tasks.rados.rados.0.smithi204.stdout:1731: left oid 4 (ObjNum 361 snap 0 seq_num 361) 2024-09-13T11:33:38.308 INFO:tasks.rados.rados.0.smithi204.stdout:1733: finishing write tid 1 to smithi20438004-24 2024-09-13T11:33:38.309 INFO:tasks.rados.rados.0.smithi204.stdout:1733: finishing write tid 2 to smithi20438004-24 2024-09-13T11:33:38.309 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:38.308+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:38.310 INFO:tasks.rados.rados.0.smithi204.stdout:1733: finishing write tid 3 to smithi20438004-24 2024-09-13T11:33:38.310 INFO:tasks.rados.rados.0.smithi204.stdout:1733: finishing write tid 4 to smithi20438004-24 2024-09-13T11:33:38.310 INFO:tasks.rados.rados.0.smithi204.stdout:1733: oid 24 updating version 0 to 130 2024-09-13T11:33:38.311 INFO:tasks.rados.rados.0.smithi204.stdout:1733: oid 24 version 130 is already newer than 129 2024-09-13T11:33:38.311 INFO:tasks.rados.rados.0.smithi204.stdout:1733: oid 24 version 130 is already newer than 128 2024-09-13T11:33:38.311 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 130 (ObjNum 363 snap 0 seq_num 363) dirty exists 2024-09-13T11:33:38.311 INFO:tasks.rados.rados.0.smithi204.stdout:1733: left oid 24 (ObjNum 363 snap 0 seq_num 363) 2024-09-13T11:33:38.311 INFO:tasks.rados.rados.0.smithi204.stdout:1734: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-13T11:33:38.356 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 386326 2024-09-13T11:33:38.387 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:38.387 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:38.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:38.409 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:38.408+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:38.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:38.432 INFO:teuthology.orchestra.run.smithi038.stderr:set 2.7 pg_upmap_items mapping to [6->0,4->7,12->3] 2024-09-13T11:33:38.432 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:33:38.459 INFO:tasks.rados.rados.0.smithi204.stdout:1731: done (3 left) 2024-09-13T11:33:38.459 INFO:tasks.rados.rados.0.smithi204.stdout:1733: done (2 left) 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1734: done (1 left) 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1735: done (0 left) 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1736: read oid 3 snap -1 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1736: expect (ObjNum 329 snap 0 seq_num 329) 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1737: tier_promote oid 38 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1738: read oid 30 snap -1 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1738: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-13T11:33:38.460 INFO:tasks.rados.rados.0.smithi204.stdout:1739: set_chunk oid 2 target oid 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:1739: set_chunk oid 2 offset: 32256 length: 123904 target oid 200608c94d52d33ff86b8f4db28451752eeae7c70062488f380f112e11b4350a offset: 0 snap -1 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:1740: read oid 37 snap -1 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:1740: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:1741: tier_flush oid 35 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:1741: tier_flush oid 35 snap -1 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:1737: got (0) Success 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 142 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-09-13T11:33:38.471 INFO:tasks.rados.rados.0.smithi204.stdout:1738: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-13T11:33:38.504 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:38.505 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:38.507 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:38.510 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:38.509+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:38.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:38.554 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:38.567 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:38.567 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:38.585 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:38.586 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:38.586 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:38.600 INFO:tasks.rados.rados.0.smithi204.stdout:1739: finishing set_chunk to oid 2 2024-09-13T11:33:38.601 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 32256 length: 123904 2024-09-13T11:33:38.601 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-09-13T11:33:38.601 INFO:tasks.rados.rados.0.smithi204.stdout:1736: expect (ObjNum 329 snap 0 seq_num 329) 2024-09-13T11:33:38.603 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:38.603 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:38.611 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:38.609+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:38.619 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:38.619 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:38.619 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:38.619 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:38.619 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:38.620 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:38.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:38.637 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:38.637 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:38.645 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2929971 2024-09-13T11:33:38.653 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:38.670 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:38.687 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:38.687 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:38.704 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:38.712 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:38.711+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:38.722 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:38.722 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:38.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:38.749 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:38.749 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:38.749 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:38.749 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:38.749 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:38.756 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:38.756 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:38.758 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:38.774 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:38.774 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:38.790 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:38.791 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:38.794 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:38.830 INFO:tasks.rados.rados.0.smithi204.stdout:1740: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:38.836 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:38.834+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:38.844 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:38.870 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1203407 2024-09-13T11:33:38.883 INFO:tasks.rados.rados.0.smithi204.stdout:1741: got (0) Success 2024-09-13T11:33:38.883 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-09-13T11:33:38.883 INFO:tasks.rados.rados.0.smithi204.stdout:1736: done (5 left) 2024-09-13T11:33:38.883 INFO:tasks.rados.rados.0.smithi204.stdout:1737: done (4 left) 2024-09-13T11:33:38.883 INFO:tasks.rados.rados.0.smithi204.stdout:1738: done (3 left) 2024-09-13T11:33:38.884 INFO:tasks.rados.rados.0.smithi204.stdout:1739: done (2 left) 2024-09-13T11:33:38.884 INFO:tasks.rados.rados.0.smithi204.stdout:1740: done (1 left) 2024-09-13T11:33:38.884 INFO:tasks.rados.rados.0.smithi204.stdout:1741: done (0 left) 2024-09-13T11:33:38.884 INFO:tasks.rados.rados.0.smithi204.stdout:1742: write (excl) oid 39 current snap is 0 2024-09-13T11:33:38.884 INFO:tasks.rados.rados.0.smithi204.stdout:1742: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-13T11:33:38.889 INFO:tasks.rados.rados.0.smithi204.stdout:1742: writing smithi20438004-39 from 534555 to 1185384 tid 1 2024-09-13T11:33:38.895 INFO:tasks.rados.rados.0.smithi204.stdout:1742: writing smithi20438004-39 from 1659104 to 2134555 tid 2 2024-09-13T11:33:38.895 INFO:tasks.rados.rados.0.smithi204.stdout:1743: read oid 29 snap -1 2024-09-13T11:33:38.895 INFO:tasks.rados.rados.0.smithi204.stdout:1743: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:38.895 INFO:tasks.rados.rados.0.smithi204.stdout:1744: set_chunk oid 10 target oid 2024-09-13T11:33:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:1744: set_chunk oid 10 offset: 911360 length: 1174016 target oid 56892569e3c134e334aa4431db07604256fc89a0ea37520d6831eed85e149585 offset: 0 snap -1 2024-09-13T11:33:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:1745: write (excl) oid 2 current snap is 0 2024-09-13T11:33:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:1742: finishing write tid 1 to smithi20438004-39 2024-09-13T11:33:38.906 INFO:tasks.rados.rados.0.smithi204.stdout:1745: seq_num 366 ranges {551362=726078,1717586=409779} 2024-09-13T11:33:38.907 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:38.907 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:38.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.549654817000004, 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.909 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:33:38.909 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:38.909 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:33:38.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:33:38.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 71.801123896000007, 2024-09-13T11:33:38.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:38.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:33:38.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 71.788906302000001, 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:33:38.919 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:33:38.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:33:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:33:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.924 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1679 4.4 4:2b094039:::smithi20438004-40:head [tier-flush] snapc 0=[] RETRY=3 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.634782195, 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0158098280000001, 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1679 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269187+0000", 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9479999999999998e-06 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269193+0000", 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.173e-06 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269209+0000", 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6039e-05 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269224+0000", 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5494000000000001e-05 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270767+0000", 2024-09-13T11:33:38.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015428869999999999 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270782+0000", 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.519e-05 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284573+0000", 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013791077000000001 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284717+0000", 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014369599999999999 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284994+0000", 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.000277324 2024-09-13T11:33:38.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2310 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.80416065, 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17044168200000001, 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.931 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2310 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099808+0000", 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100126+0000", 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031826000000000001 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100144+0000", 2024-09-13T11:33:38.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8029999999999998e-05 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100153+0000", 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0119999999999994e-06 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100230+0000", 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6221000000000001e-05 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100242+0000", 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.257e-05 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270222+0000", 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16997934100000001 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270248+0000", 2024-09-13T11:33:38.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6366000000000001e-05 2024-09-13T11:33:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.936 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2311 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.797620373999997, 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16403588, 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.937 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2311 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106347+0000", 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106673+0000", 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588799999999998 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106687+0000", 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3247e-05 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106693+0000", 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2249999999999997e-06 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106741+0000", 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8715999999999999e-05 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106750+0000", 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3159999999999997e-06 2024-09-13T11:33:38.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270375+0000", 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16362501300000001 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270382+0000", 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.941 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2312 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.792329111999997, 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158778577, 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2312 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111639+0000", 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111756+0000", 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117166 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111770+0000", 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3492999999999999e-05 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111778+0000", 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5820000000000004e-06 2024-09-13T11:33:38.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111834+0000", 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6564000000000003e-05 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111842+0000", 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8280000000000003e-06 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270410+0000", 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15856827400000001 2024-09-13T11:33:38.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270416+0000", 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2313 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.792174781, 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158679564, 2024-09-13T11:33:38.946 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2313 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:38.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6399999999999998e-07 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111795+0000", 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0450000000000001e-06 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111799+0000", 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5250000000000001e-06 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111802+0000", 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.162e-06 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111876+0000", 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4062999999999999e-05 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111885+0000", 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0850000000000003e-06 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270465+0000", 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158579839 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270472+0000", 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4810000000000003e-06 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4.8a282f38 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.792144565000001, 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15868154300000001, 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.951 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111823+0000", 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3299999999999998e-07 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111824+0000", 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7290000000000001e-06 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111829+0000", 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5859999999999998e-06 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111832+0000", 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.456e-06 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111901+0000", 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8447000000000004e-05 2024-09-13T11:33:38.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111905+0000", 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8569999999999997e-06 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270497+0000", 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158592381 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270504+0000", 2024-09-13T11:33:38.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.754e-06 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2309 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.634649447000001, 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.00703426, 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.956 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2309 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269319+0000", 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.265e-06 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269324+0000", 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1800000000000004e-06 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269334+0000", 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7310000000000005e-06 2024-09-13T11:33:38.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269343+0000", 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8339999999999994e-06 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269381+0000", 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7709000000000001e-05 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269391+0000", 2024-09-13T11:33:38.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278002+0000", 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086114859999999998 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278150+0000", 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014796199999999999 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.276352+0000", 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99820153300000003 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2170 4.12 4:4b088bf8:::smithi20438004-24:head [write 2083958~603055] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.626083585000003, 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029083859, 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.961 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2170 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277884+0000", 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4399999999999998e-07 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278124+0000", 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000239485 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278133+0000", 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3219999999999999e-06 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278141+0000", 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280144+0000", 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020033289999999999 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280155+0000", 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0966e-05 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301300+0000", 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021145557999999998 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301414+0000", 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011306099999999999 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301601+0000", 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000187357 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304518+0000", 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029169539999999998 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304561+0000", 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229000000000002e-05 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306715+0000", 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002153706 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306792+0000", 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7466999999999995e-05 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306851+0000", 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8811999999999999e-05 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306967+0000", 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116062 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2171 4.12 4:4b088bf8:::smithi20438004-24:head [setxattr _header (56) in=7b,truncate 3058874] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.625811016999997, 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029067939000000001, 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2171 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278157+0000", 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0699999999999996e-07 2024-09-13T11:33:38.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278161+0000", 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2640000000000002e-06 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278173+0000", 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1868999999999999e-05 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278179+0000", 2024-09-13T11:33:38.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280537+0000", 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023576320000000001 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280544+0000", 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7560000000000002e-06 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303651+0000", 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023107240000000001 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303802+0000", 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150774 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303982+0000", 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018009800000000001 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306699+0000", 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027165100000000001 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306720+0000", 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1535e-05 2024-09-13T11:33:38.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306753+0000", 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3019999999999999e-05 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307010+0000", 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025724800000000001 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307061+0000", 2024-09-13T11:33:38.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0633000000000001e-05 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307224+0000", 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000162843 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2172 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.625778333, 2024-09-13T11:33:38.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029398245999999999, 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2172 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000005e-07 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278193+0000", 2024-09-13T11:33:38.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278201+0000", 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278207+0000", 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4289999999999998e-06 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280568+0000", 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023610900000000002 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280577+0000", 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8259999999999999e-06 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304605+0000", 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024027829000000001 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304704+0000", 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9610000000000003e-05 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307334+0000", 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002629667 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307421+0000", 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7433000000000001e-05 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307587+0000", 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016528800000000001 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2219 4.12 4:4d4c8512:::smithi20438004-29:head [sparse-read 0~2348017] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.625707124999998, 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029874108999999999, 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.982 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2219 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3599999999999999e-07 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278262+0000", 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2110000000000001e-06 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278269+0000", 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.314e-06 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278274+0000", 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5249999999999996e-06 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280850+0000", 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025761389999999999 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280858+0000", 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8760000000000005e-06 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307628+0000", 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.026769465999999999 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307703+0000", 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4847999999999999e-05 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.308134+0000", 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00043129399999999998 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.623951658000003, 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029229465, 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.987 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280016+0000", 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280020+0000", 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280026+0000", 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1070000000000001e-06 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280032+0000", 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280109+0000", 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6298999999999996e-05 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280116+0000", 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5129999999999999e-06 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298271+0000", 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018154805 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298334+0000", 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3268999999999993e-05 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309030+0000", 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010695566 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309089+0000", 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309245+0000", 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000155317 2024-09-13T11:33:38.992 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2340 4.18 4:1d367c64:::smithi20438004-45:head [write 578207~550227] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.623923910999999, 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030475704999999999, 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:38.993 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2340 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0399999999999996e-07 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280452+0000", 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040812800000000001 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280461+0000", 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3829999999999998e-06 2024-09-13T11:33:38.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280468+0000", 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0400000000000004e-06 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280506+0000", 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8037000000000002e-05 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280514+0000", 2024-09-13T11:33:38.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.011e-06 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298673+0000", 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018159411 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298829+0000", 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015594 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:38.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.299023+0000", 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193838 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307549+0000", 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085256289999999998 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307635+0000", 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6596999999999998e-05 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307689+0000", 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4027999999999998e-05 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307748+0000", 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8733999999999997e-05 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310255+0000", 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025070940000000001 2024-09-13T11:33:38.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310375+0000", 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119691 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310519+0000", 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014364 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2341 4.18 4:1d367c64:::smithi20438004-45:head [write 1762410~779734] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.623481544000001, 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031687067999999999, 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2341 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280486+0000", 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281018+0000", 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053231899999999996 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281027+0000", 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.704e-06 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281034+0000", 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0659999999999998e-06 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281159+0000", 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124661 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281168+0000", 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000001e-06 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302725+0000", 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021557478000000001 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302889+0000", 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016421299999999999 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303087+0000", 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019817200000000001 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307806+0000", 2024-09-13T11:33:39.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004718168 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307868+0000", 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2775999999999997e-05 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307920+0000", 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1749000000000002e-05 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307971+0000", 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1343e-05 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311829+0000", 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038571769999999998 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312043+0000", 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021412199999999999 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312172+0000", 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012966900000000001 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2342 4.18 4:1d367c64:::smithi20438004-45:head [write 3286845~91362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.622914393000002, 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031505618999999999, 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.008 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2342 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281053+0000", 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2799999999999995e-07 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281104+0000", 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0550999999999998e-05 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281112+0000", 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0129999999999993e-06 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281118+0000", 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281191+0000", 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2738999999999996e-05 2024-09-13T11:33:39.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281197+0000", 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309351+0000", 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028153293999999999 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309441+0000", 2024-09-13T11:33:39.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0048000000000002e-05 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309594+0000", 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153609 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311315+0000", 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017209829999999999 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311795+0000", 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048009300000000001 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312281+0000", 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048612699999999999 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312394+0000", 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112935 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312404+0000", 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0112000000000001e-05 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312449+0000", 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4650000000000001e-05 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312558+0000", 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108959 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.015 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2343 4.18 4:1d367c64:::smithi20438004-45:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.622838842999997, 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031723945000000003, 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.016 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2343 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281129+0000", 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281132+0000", 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.988e-06 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281137+0000", 2024-09-13T11:33:39.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.95e-06 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281142+0000", 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4e-06 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281219+0000", 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6946000000000005e-05 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281225+0000", 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.601e-06 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310552+0000", 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029326497 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310643+0000", 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1416999999999996e-05 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310817+0000", 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000173463 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312229+0000", 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001412424 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312324+0000", 2024-09-13T11:33:39.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4264000000000004e-05 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312578+0000", 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025401699999999999 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312605+0000", 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7557000000000001e-05 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312650+0000", 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4496000000000002e-05 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312683+0000", 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2898000000000002e-05 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312852+0000", 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000169436 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.023 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2344 4.18 4:1d367c64:::smithi20438004-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.622819096000001, 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032021714999999999, 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.024 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2344 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.15e-07 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281150+0000", 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.857e-06 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281155+0000", 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0579999999999997e-06 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281160+0000", 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0409999999999997e-06 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281248+0000", 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7979000000000006e-05 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281255+0000", 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5089999999999999e-06 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.027 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311169+0000", 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02991394 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311266+0000", 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7007000000000001e-05 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312894+0000", 2024-09-13T11:33:39.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016280489999999999 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312976+0000", 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2052999999999995e-05 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.313169+0000", 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193807 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.029 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.030 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.030 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:39.031 INFO:tasks.rados.rados.0.smithi204.stdout:1745: writing smithi20438004-2 from 551362 to 1277440 tid 1 2024-09-13T11:33:39.031 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:39.034 INFO:tasks.rados.rados.0.smithi204.stdout:1745: writing smithi20438004-2 from 1717586 to 2127365 tid 2 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1746: read oid 11 snap -1 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1746: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1747: write (excl) oid 15 current snap is 0 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1747: seq_num 367 ranges {568169=401327,1376068=669790} 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1747: writing smithi20438004-15 from 568169 to 969496 tid 1 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1747: writing smithi20438004-15 from 1376068 to 2045858 tid 2 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1748: read oid 23 snap -1 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1748: expect deleted 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1749: read oid 49 snap -1 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1749: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1750: write oid 46 current snap is 0 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1742: finishing write tid 2 to smithi20438004-39 2024-09-13T11:33:39.035 INFO:tasks.rados.rados.0.smithi204.stdout:1750: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1750: writing smithi20438004-46 from 584976 to 1061552 tid 1 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1750: writing smithi20438004-46 from 1834550 to 2184976 tid 2 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1751: set_chunk oid 34 target oid 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1751: set_chunk oid 34 offset: 3357696 length: 155136 target oid 00c92e678daa60988c0cbc4f4fdc35deae9741a9ef20a27c365e43c3477f2199 offset: 0 snap -1 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1752: write oid 31 current snap is 0 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1742: finishing write tid 3 to smithi20438004-39 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1742: finishing write tid 4 to smithi20438004-39 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1742: oid 39 updating version 0 to 123 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1742: oid 39 version 123 is already newer than 122 2024-09-13T11:33:39.036 INFO:tasks.rados.rados.0.smithi204.stdout:1742: oid 39 updating version 123 to 124 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 124 (ObjNum 365 snap 0 seq_num 365) dirty exists 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1742: left oid 39 (ObjNum 365 snap 0 seq_num 365) 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1752: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1752: writing smithi20438004-31 from 601783 to 1153608 tid 1 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1752: writing smithi20438004-31 from 1776679 to 2201783 tid 2 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1742: done (10 left) 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1753: set_chunk oid 50 target oid 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1753: set_chunk oid 50 offset: 1030144 length: 1351680 target oid c27de93a185f65c9b00169a0ecfc789beba4015186542755a4fcc06d5686514e offset: 0 snap -1 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1754: read oid 3 snap -1 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1754: expect (ObjNum 329 snap 0 seq_num 329) 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1755: read oid 28 snap -1 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1755: expect (ObjNum 288 snap 0 seq_num 288) 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1756: set_chunk oid 8 target oid 2024-09-13T11:33:39.037 INFO:tasks.rados.rados.0.smithi204.stdout:1756: set_chunk oid 8 offset: 828416 length: 945152 target oid 622747d72f42ef643ce5cef4a3a69bbf70e16aa477447eeadce5daa3fa5c12f4 offset: 0 snap -1 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:1757: read oid 12 snap -1 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:1757: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:1758: read oid 40 snap -1 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:1758: expect deleted 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:1744: finishing set_chunk to oid 10 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 911360 length: 1174016 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 0 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-13T11:33:39.038 INFO:tasks.rados.rados.0.smithi204.stdout:1743: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:39.038 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:39.029+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:39.054 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:39.131 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:39.130+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:39.158 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:39.159 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:39.159 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:39.159 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:39.167 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:39.191 INFO:tasks.rados.rados.0.smithi204.stdout:1745: finishing write tid 1 to smithi20438004-2 2024-09-13T11:33:39.191 INFO:tasks.rados.rados.0.smithi204.stdout:1746: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:39.233 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:39.231+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:39.271 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:39.272 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:39.272 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:39.272 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:39.272 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:39.280 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:39.364 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:39.363+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:39.420 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2383998 2024-09-13T11:33:39.426 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:39.426 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.326670093999994, 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.314235948999993, 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.432 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:33:39.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.304282354999998, 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:33:39.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:33:39.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:33:39.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.273564547999996, 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:33:39.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:33:39.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.447 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.254741229000004, 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:33:39.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.241221879999998, 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.452 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:33:39.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.456 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.231581547000005, 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:33:39.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:33:39.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.221276047000003, 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:39.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:33:39.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.210890105000004, 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:33:39.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:36 3.3 3:e2816a99:::d4bb749789c90c884ffd20ac62f804c7a6b80502d14432fa627e1239249ee2da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:39.470 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.153362706000003, 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013305613000000001, 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:39.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270480+0000", 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2399999999999998e-07 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270655+0000", 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017481300000000001 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270683+0000", 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8340999999999999e-05 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270693+0000", 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271029+0000", 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033640200000000002 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271038+0000", 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278355+0000", 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073166510000000004 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278481+0000", 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125955 2024-09-13T11:33:39.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278657+0000", 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176816 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280615+0000", 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019576810000000002 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283538+0000", 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029229030000000001 2024-09-13T11:33:39.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283592+0000", 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4366000000000001e-05 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283671+0000", 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8337000000000006e-05 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283785+0000", 2024-09-13T11:33:39.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114412 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:40 3.3 3:c7a12847:::768a08ae37e9bf19369a517906ebd209c1f2b8ad3684596d644da7d48b42114d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.153119599999997, 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013347114, 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270839+0000", 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115369 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270847+0000", 2024-09-13T11:33:39.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270854+0000", 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271073+0000", 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021964400000000001 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271080+0000", 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7909999999999999e-06 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278928+0000", 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078479819999999999 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279058+0000", 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130102 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279300+0000", 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024166300000000001 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280641+0000", 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013409229999999999 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283545+0000", 2024-09-13T11:33:39.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002904401 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283927+0000", 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038193199999999997 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283964+0000", 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7117999999999999e-05 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284070+0000", 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010545899999999999 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:41 3.3 3:e9fa9a15:::3c65c6c9770b7a29cd5a9b6dc5b0b39ab23cb2a64bfb19442efcb911d255e067:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.152975986999998, 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013499547000000001, 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.484 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270867+0000", 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270893+0000", 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6057000000000001e-05 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270901+0000", 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270915+0000", 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4450000000000001e-05 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271104+0000", 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018904600000000001 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271111+0000", 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6039999999999998e-06 2024-09-13T11:33:39.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280674+0000", 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095629140000000005 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280811+0000", 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136802 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:39.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281027+0000", 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000216617 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283802+0000", 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027746540000000001 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283884+0000", 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2167000000000001e-05 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284178+0000", 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029347899999999997 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284246+0000", 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8609999999999995e-05 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284366+0000", 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011950799999999999 2024-09-13T11:33:39.490 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:43 3.3 3:dc8816b1:::1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.152915292000003, 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013665096999999999, 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270933+0000", 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2410000000000003e-06 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270939+0000", 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5810000000000002e-06 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270945+0000", 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:33:39.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271134+0000", 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000189161 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271140+0000", 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2999999999999998e-06 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281407+0000", 2024-09-13T11:33:39.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010266226 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281549+0000", 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142583 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281737+0000", 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018791500000000001 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283810+0000", 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002072571 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284127+0000", 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000317516 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284417+0000", 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029000899999999998 2024-09-13T11:33:39.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284480+0000", 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3136e-05 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284592+0000", 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111712 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 43.57908742, 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43316489499999999, 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.498 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844756+0000", 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844760+0000", 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0550000000000001e-06 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844769+0000", 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6129999999999993e-06 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844776+0000", 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0280000000000003e-06 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7639000000000002e-05 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844841+0000", 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1049999999999997e-06 2024-09-13T11:33:39.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277896+0000", 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43305493499999997 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277920+0000", 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3614000000000001e-05 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 43.579035787000002, 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43315226600000001, 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.503 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844807+0000", 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3200000000000003e-07 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844811+0000", 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.799e-06 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844827+0000", 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5461999999999999e-05 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.126e-06 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5369999999999999e-05 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844863+0000", 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.658e-06 2024-09-13T11:33:39.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277951+0000", 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308847900000003 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277959+0000", 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 43.578994076000001, 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43314228399999999, 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844849+0000", 2024-09-13T11:33:39.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0100000000000005e-07 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844853+0000", 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9589999999999997e-06 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4110000000000001e-06 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844864+0000", 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8930000000000004e-06 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844892+0000", 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8042e-05 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844898+0000", 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5049999999999998e-06 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277984+0000", 2024-09-13T11:33:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308647700000003 2024-09-13T11:33:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277990+0000", 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3960000000000003e-06 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4:8c30e60e:::smithi20438004-18:head [read 0~2487436,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.141444753999998, 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012791258999999999, 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.513 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282399+0000", 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.691e-06 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282404+0000", 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.887e-06 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282414+0000", 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9539999999999999e-06 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282423+0000", 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6039999999999993e-06 2024-09-13T11:33:39.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282477+0000", 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4738999999999999e-05 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282486+0000", 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5450000000000003e-06 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287493+0000", 2024-09-13T11:33:39.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050068949999999999 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287636+0000", 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014333299999999999 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.295189+0000", 2024-09-13T11:33:39.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075526109999999999 2024-09-13T11:33:39.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.520 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 153896~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.141343163999998, 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014013433, 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.521 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282500+0000", 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5499999999999998e-07 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282503+0000", 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.563e-06 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282510+0000", 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8820000000000003e-06 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282516+0000", 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282562+0000", 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5868999999999997e-05 2024-09-13T11:33:39.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282570+0000", 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296085+0000", 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013515255 2024-09-13T11:33:39.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296187+0000", 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101457 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296512+0000", 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588099999999998 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 447518~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 42.141258671000003, 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014496826000000001, 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.527 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5000000000000002e-07 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282589+0000", 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4220000000000002e-06 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282595+0000", 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8789999999999997e-06 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282606+0000", 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0229e-05 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282652+0000", 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6610999999999998e-05 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282661+0000", 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2640000000000008e-06 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296550+0000", 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013888981999999999 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296628+0000", 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8564000000000002e-05 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.297080+0000", 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000452225 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3902 4.11 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:31.072365+0000", 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 8.3514769330000007, 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011701236, 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.533 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3902 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072365+0000", 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072365+0000", 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072367+0000", 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.668e-06 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072392+0000", 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5358e-05 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072407+0000", 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4669e-05 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072415+0000", 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8439999999999998e-06 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072475+0000", 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0424000000000003e-05 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072568+0000", 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3255000000000001e-05 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.084066+0000", 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011498018 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.537 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:39.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: finishing write tid 2 to smithi20438004-2 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: finishing write tid 3 to smithi20438004-2 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: finishing write tid 4 to smithi20438004-2 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: oid 2 updating version 0 to 93 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: oid 2 updating version 93 to 94 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: oid 2 version 94 is already newer than 92 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 94 (ObjNum 366 snap 0 seq_num 366) dirty exists 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: left oid 2 (ObjNum 366 snap 0 seq_num 366) 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1747: finishing write tid 1 to smithi20438004-15 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1743: done (15 left) 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1744: done (14 left) 2024-09-13T11:33:39.541 INFO:tasks.rados.rados.0.smithi204.stdout:1745: done (13 left) 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1746: done (12 left) 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1759: read oid 41 snap -1 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1759: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1760: read oid 37 snap -1 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1760: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1761: write oid 30 current snap is 0 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1747: finishing write tid 2 to smithi20438004-15 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1747: finishing write tid 3 to smithi20438004-15 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:1749: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-13T11:33:39.542 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:39.542 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:39.537+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:39.544 INFO:tasks.rados.rados.0.smithi204.stdout:1747: finishing write tid 4 to smithi20438004-15 2024-09-13T11:33:39.544 INFO:tasks.rados.rados.0.smithi204.stdout:1747: oid 15 updating version 0 to 113 2024-09-13T11:33:39.544 INFO:tasks.rados.rados.0.smithi204.stdout:1747: oid 15 version 113 is already newer than 111 2024-09-13T11:33:39.544 INFO:tasks.rados.rados.0.smithi204.stdout:1747: oid 15 version 113 is already newer than 112 2024-09-13T11:33:39.544 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 113 (ObjNum 367 snap 0 seq_num 367) dirty exists 2024-09-13T11:33:39.545 INFO:tasks.rados.rados.0.smithi204.stdout:1747: left oid 15 (ObjNum 367 snap 0 seq_num 367) 2024-09-13T11:33:39.545 INFO:tasks.rados.rados.0.smithi204.stdout:1750: finishing write tid 1 to smithi20438004-46 2024-09-13T11:33:39.545 INFO:tasks.rados.rados.0.smithi204.stdout:1761: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-13T11:33:39.551 INFO:tasks.rados.rados.0.smithi204.stdout:1761: writing smithi20438004-30 from 618590 to 1245664 tid 1 2024-09-13T11:33:39.557 INFO:tasks.rados.rados.0.smithi204.stdout:1761: writing smithi20438004-30 from 1835161 to 2218590 tid 2 2024-09-13T11:33:39.557 INFO:tasks.rados.rados.0.smithi204.stdout:1747: done (14 left) 2024-09-13T11:33:39.557 INFO:tasks.rados.rados.0.smithi204.stdout:1748: done (13 left) 2024-09-13T11:33:39.557 INFO:tasks.rados.rados.0.smithi204.stdout:1749: done (12 left) 2024-09-13T11:33:39.557 INFO:tasks.rados.rados.0.smithi204.stdout:1762: write oid 9 current snap is 0 2024-09-13T11:33:39.557 INFO:tasks.rados.rados.0.smithi204.stdout:1762: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-13T11:33:39.564 INFO:tasks.rados.rados.0.smithi204.stdout:1762: writing smithi20438004-9 from 635397 to 1337720 tid 1 2024-09-13T11:33:39.569 INFO:tasks.rados.rados.0.smithi204.stdout:1762: writing smithi20438004-9 from 1893643 to 2235397 tid 2 2024-09-13T11:33:39.569 INFO:tasks.rados.rados.0.smithi204.stdout:1763: write oid 21 current snap is 0 2024-09-13T11:33:39.569 INFO:tasks.rados.rados.0.smithi204.stdout:1750: finishing write tid 2 to smithi20438004-46 2024-09-13T11:33:39.569 INFO:tasks.rados.rados.0.smithi204.stdout:1751: finishing set_chunk to oid 34 2024-09-13T11:33:39.569 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 134 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-09-13T11:33:39.569 INFO:tasks.rados.rados.0.smithi204.stdout:1750: finishing write tid 3 to smithi20438004-46 2024-09-13T11:33:39.569 INFO:tasks.rados.rados.0.smithi204.stdout:1763: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-13T11:33:39.577 INFO:tasks.rados.rados.0.smithi204.stdout:1763: writing smithi20438004-21 from 652204 to 1429776 tid 1 2024-09-13T11:33:39.582 INFO:tasks.rados.rados.0.smithi204.stdout:1763: writing smithi20438004-21 from 1952125 to 2252204 tid 2 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1751: done (13 left) 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1764: read oid 16 snap -1 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1764: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1765: read oid 25 snap -1 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1765: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1766: read oid 4 snap -1 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1766: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1750: finishing write tid 4 to smithi20438004-46 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1750: oid 46 updating version 0 to 100 2024-09-13T11:33:39.583 INFO:tasks.rados.rados.0.smithi204.stdout:1750: oid 46 version 100 is already newer than 98 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1750: oid 46 version 100 is already newer than 99 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 100 (ObjNum 368 snap 0 seq_num 368) dirty exists 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1750: left oid 46 (ObjNum 368 snap 0 seq_num 368) 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1752: finishing write tid 1 to smithi20438004-31 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1752: finishing write tid 2 to smithi20438004-31 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1750: done (15 left) 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1767: read oid 46 snap -1 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1767: expect (ObjNum 368 snap 0 seq_num 368) 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1752: finishing write tid 3 to smithi20438004-31 2024-09-13T11:33:39.584 INFO:tasks.rados.rados.0.smithi204.stdout:1752: finishing write tid 4 to smithi20438004-31 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:1752: oid 31 updating version 0 to 60 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:1752: oid 31 updating version 60 to 62 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:1752: oid 31 version 62 is already newer than 61 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 62 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:1752: left oid 31 (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:1753: finishing set_chunk to oid 50 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 72 (ObjNum 350 snap 0 seq_num 350) dirty exists 2024-09-13T11:33:39.585 INFO:tasks.rados.rados.0.smithi204.stdout:1754: expect (ObjNum 329 snap 0 seq_num 329) 2024-09-13T11:33:39.639 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:39.638+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:39.653 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:39.653 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:39.654 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:39.654 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:39.662 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:39.740 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:39.738+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:39.803 INFO:tasks.rados.rados.0.smithi204.stdout:1755: expect (ObjNum 288 snap 0 seq_num 288) 2024-09-13T11:33:39.803 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:39.806 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:39.806 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:39.806 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:39.806 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:39.806 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:39.815 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:39.841 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:39.840+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:39.920 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:39.920 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:39.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.816238827999996, 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016706373, 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.921 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.922 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.99e-07 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104537+0000", 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034575700000000001 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104552+0000", 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5475999999999999e-05 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104566+0000", 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3766e-05 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104623+0000", 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7046999999999998e-05 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104650+0000", 2024-09-13T11:33:39.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6786000000000001e-05 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117752+0000", 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013102387 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117785+0000", 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2741999999999999e-05 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.812279501999996, 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025912691000000002, 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:39.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105039+0000", 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7100000000000001e-07 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108050+0000", 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030115910000000001 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108061+0000", 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0512e-05 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108069+0000", 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4479999999999994e-06 2024-09-13T11:33:39.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108126+0000", 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7024999999999997e-05 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108165+0000", 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8701000000000001e-05 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130916+0000", 2024-09-13T11:33:39.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022750829 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130951+0000", 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4913999999999998e-05 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:203 3.1 e38/30 3:970ba768:::smithi20438004-4 low_tier:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.809073009000002, 2024-09-13T11:33:39.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031827052000000002, 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:39.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108245+0000", 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1800000000000005e-07 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111202+0000", 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029571440000000001 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111221+0000", 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8887000000000001e-05 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111248+0000", 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6458e-05 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111707+0000", 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045935799999999999 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111729+0000", 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140022+0000", 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028292571999999998 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140071+0000", 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9669000000000002e-05 2024-09-13T11:33:39.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.806049458999993, 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028930938999999999, 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.935 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111269+0000", 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5700000000000002e-07 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113854+0000", 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025854340000000002 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113866+0000", 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1884e-05 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113873+0000", 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116175+0000", 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023019 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116218+0000", 2024-09-13T11:33:39.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2796000000000002e-05 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140165+0000", 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023947718999999999 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140199+0000", 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3596000000000002e-05 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:218 3.1 e38/30 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.796186551000005, 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025863384999999999, 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121132+0000", 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3260000000000001e-06 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124734+0000", 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036019379999999998 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124748+0000", 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913999999999999e-05 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124760+0000", 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1786e-05 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124849+0000", 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8882999999999996e-05 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124874+0000", 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5466999999999999e-05 2024-09-13T11:33:39.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146945+0000", 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022070837999999999 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146994+0000", 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9233000000000001e-05 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.944 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.789253368000004, 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020872166000000001, 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128066+0000", 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.86e-06 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131456+0000", 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033901360000000002 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131467+0000", 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0806e-05 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131474+0000", 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131524+0000", 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9262999999999999e-05 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131554+0000", 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0148999999999999e-05 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148883+0000", 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017329445999999998 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148936+0000", 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2879e-05 2024-09-13T11:33:39.949 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.785815169000003, 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017580887, 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131503+0000", 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:33:39.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133995+0000", 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024920630000000001 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134006+0000", 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0696000000000001e-05 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134012+0000", 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136541+0000", 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025286150000000001 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136563+0000", 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2531000000000002e-05 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149031+0000", 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012467951 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149083+0000", 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2040999999999999e-05 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:39.954 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.759606590999994, 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022274233000000001, 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157712+0000", 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3689999999999999e-06 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161974+0000", 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042619160000000001 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161990+0000", 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5506e-05 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161998+0000", 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7520000000000002e-06 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162103+0000", 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104415 2024-09-13T11:33:39.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162126+0000", 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3643e-05 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179932+0000", 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017805476000000001 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179985+0000", 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3155999999999997e-05 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.755281358000005, 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018684003000000001, 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162037+0000", 2024-09-13T11:33:39.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.083e-06 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166161+0000", 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041238799999999999 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166179+0000", 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8329999999999999e-05 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166190+0000", 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0822000000000001e-05 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166302+0000", 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011233700000000001 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166328+0000", 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5185999999999999e-05 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180646+0000", 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014318261000000001 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180720+0000", 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.963 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.751102356000004, 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014657148, 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1300000000000001e-07 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169754+0000", 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035389779999999999 2024-09-13T11:33:39.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169772+0000", 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8020999999999999e-05 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0037999999999999e-05 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169847+0000", 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4695000000000003e-05 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169865+0000", 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7365999999999999e-05 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180834+0000", 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010969016 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180872+0000", 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8621000000000002e-05 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:39.968 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.739191203000004, 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018123856000000001, 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178128+0000", 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181360+0000", 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032324699999999999 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181371+0000", 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0978000000000001e-05 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181378+0000", 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.861e-06 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181431+0000", 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3173000000000001e-05 2024-09-13T11:33:39.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181455+0000", 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4236999999999999e-05 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196213+0000", 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01475807 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196250+0000", 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6636e-05 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.735907646000001, 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016397140000000001, 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.973 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181411+0000", 2024-09-13T11:33:39.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7399999999999991e-07 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184468+0000", 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030575759999999998 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184481+0000", 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3072e-05 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184491+0000", 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184550+0000", 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8251999999999997e-05 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184575+0000", 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689000000000001e-05 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197720+0000", 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013144708999999999 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197807+0000", 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6799999999999996e-05 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.977 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:281 4.a 4:56ef4fd4:::smithi20438004-32:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.684868933999994, 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018358335999999999, 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232449+0000", 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7499999999999998e-07 2024-09-13T11:33:39.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232457+0000", 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3610000000000003e-06 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232470+0000", 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3230999999999999e-05 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232478+0000", 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5989999999999996e-06 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232541+0000", 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3717e-05 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232619+0000", 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248106+0000", 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015486616 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248991+0000", 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088509199999999998 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249363+0000", 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037176599999999999 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250719+0000", 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001356255 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250765+0000", 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6053000000000001e-05 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250807+0000", 2024-09-13T11:33:39.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1675000000000003e-05 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1696 4.b 4:da92e835:::smithi20438004-50:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.648717118999997, 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016782392, 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.984 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1696 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268602+0000", 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268608+0000", 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7629999999999997e-06 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268620+0000", 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1950000000000001e-05 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268632+0000", 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2330999999999999e-05 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269210+0000", 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057761900000000003 2024-09-13T11:33:39.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269221+0000", 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.168e-05 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277483+0000", 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0082621180000000006 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277641+0000", 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000157365 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285383+0000", 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00774195 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1726 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:39.989 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.648648661999999, 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015213417999999999, 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1726 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:39.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.58e-07 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268673+0000", 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7809999999999999e-06 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268679+0000", 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7729999999999996e-06 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268684+0000", 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-13T11:33:39.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269275+0000", 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059124500000000001 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269281+0000", 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2779999999999999e-06 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281010+0000", 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011728762 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281114+0000", 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103872 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283882+0000", 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027680539999999998 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:39.994 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1727 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.648626915000001, 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016698689999999999, 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1727 2024-09-13T11:33:39.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:39.996 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268691+0000", 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6100000000000001e-07 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268694+0000", 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.672e-06 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268699+0000", 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1819999999999997e-06 2024-09-13T11:33:39.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268703+0000", 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269302+0000", 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059838199999999995 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269307+0000", 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3170000000000001e-06 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284218+0000", 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014911308999999999 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284330+0000", 2024-09-13T11:33:39.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111715 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285389+0000", 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010592729999999999 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2018 4.5 4:a9b0c9bd:::smithi20438004-37:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.041659238000001, 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021048396, 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2018 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:40.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875659+0000", 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875661+0000", 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9920000000000002e-06 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875665+0000", 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875668+0000", 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875714+0000", 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5189999999999999e-05 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875770+0000", 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6824999999999999e-05 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.896707+0000", 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020936156000000001 2024-09-13T11:33:40.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3496 4.18 4:1cf41451:::smithi20438004-7:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 16.163293315000001, 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018591032, 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.005 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3496 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754026+0000", 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4610000000000001e-06 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754033+0000", 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0369999999999997e-06 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754047+0000", 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4063000000000001e-05 2024-09-13T11:33:40.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754056+0000", 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5980000000000003e-06 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754109+0000", 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2599999999999998e-05 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754165+0000", 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6079999999999998e-05 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.772615+0000", 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018450193 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4005 4.a 4:56ef4fd4:::smithi20438004-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e81)", 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 5.23764495, 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018951355999999999, 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4005 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.010 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.011 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679675+0000", 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2079999999999999e-06 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679682+0000", 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3839999999999999e-06 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679701+0000", 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865999999999999e-05 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679710+0000", 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1980000000000007e-06 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679782+0000", 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2163999999999996e-05 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679869+0000", 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6366000000000006e-05 2024-09-13T11:33:40.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.698624+0000", 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018755170000000002 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4055 4.b 4:d738a69d:::smithi20438004-16:head [write 400099~616131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 4.3191364800000001, 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01451794, 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.014 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4055 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598185+0000", 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4120000000000003e-06 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598800+0000", 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061453600000000005 2024-09-13T11:33:40.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598830+0000", 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0593999999999999e-05 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598847+0000", 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6552e-05 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598988+0000", 2024-09-13T11:33:40.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014121000000000001 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.599001+0000", 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2605e-05 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603641+0000", 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046401250000000002 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.018 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603745+0000", 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010398700000000001 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603921+0000", 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017563 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.607101+0000", 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031806360000000001 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.611101+0000", 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039995650000000001 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612503+0000", 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014018909999999999 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612653+0000", 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015040499999999999 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612699+0000", 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5791999999999998e-05 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.021 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:40.022 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:40.023 INFO:tasks.rados.rados.0.smithi204.stdout:1756: finishing set_chunk to oid 8 2024-09-13T11:33:40.024 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 828416 length: 945152 2024-09-13T11:33:40.024 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-13T11:33:40.024 INFO:tasks.rados.rados.0.smithi204.stdout:1757: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-13T11:33:40.024 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 686655 2024-09-13T11:33:40.025 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:40.024+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:40.053 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 849769 2024-09-13T11:33:40.112 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.126 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:40.124+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:40.127 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:40.127 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:40.127 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:40.127 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:40.135 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:40.182 INFO:tasks.rados.rados.0.smithi204.stdout:1759: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-13T11:33:40.186 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1960008 2024-09-13T11:33:40.204 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.227 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:40.225+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:40.264 INFO:tasks.rados.rados.0.smithi204.stdout:1760: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:40.281 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:40.281 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:40.281 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:40.281 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:40.281 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:40.289 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:40.308 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.333 INFO:tasks.rados.rados.0.smithi204.stdout:1761: finishing write tid 1 to smithi20438004-30 2024-09-13T11:33:40.333 INFO:tasks.rados.rados.0.smithi204.stdout:1761: finishing write tid 2 to smithi20438004-30 2024-09-13T11:33:40.333 INFO:tasks.rados.rados.0.smithi204.stdout:1752: done (15 left) 2024-09-13T11:33:40.333 INFO:tasks.rados.rados.0.smithi204.stdout:1753: done (14 left) 2024-09-13T11:33:40.333 INFO:tasks.rados.rados.0.smithi204.stdout:1754: done (13 left) 2024-09-13T11:33:40.333 INFO:tasks.rados.rados.0.smithi204.stdout:1755: done (12 left) 2024-09-13T11:33:40.334 INFO:tasks.rados.rados.0.smithi204.stdout:1756: done (11 left) 2024-09-13T11:33:40.334 INFO:tasks.rados.rados.0.smithi204.stdout:1757: done (10 left) 2024-09-13T11:33:40.334 INFO:tasks.rados.rados.0.smithi204.stdout:1758: done (9 left) 2024-09-13T11:33:40.334 INFO:tasks.rados.rados.0.smithi204.stdout:1759: done (8 left) 2024-09-13T11:33:40.334 INFO:tasks.rados.rados.0.smithi204.stdout:1760: done (7 left) 2024-09-13T11:33:40.334 INFO:tasks.rados.rados.0.smithi204.stdout:1768: set_chunk oid 43 target oid 2024-09-13T11:33:40.334 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:40.332+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:40.337 INFO:tasks.rados.rados.0.smithi204.stdout:1768: set_chunk oid 43 offset: 784384 length: 359424 target oid 3ce4dfe8be6134840f7280c7c3b9652339978401af9b5c0001a86746ed4cf8f6 offset: 0 snap -1 2024-09-13T11:33:40.337 INFO:tasks.rados.rados.0.smithi204.stdout:1769: read oid 32 snap -1 2024-09-13T11:33:40.338 INFO:tasks.rados.rados.0.smithi204.stdout:1769: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:40.338 INFO:tasks.rados.rados.0.smithi204.stdout:1770: set_chunk oid 5 target oid 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1770: set_chunk oid 5 offset: 3256320 length: 203264 target oid e6b635eef135ddf4e43989c8a10ebd0bcf368dd2cacdc15ec76a4dbedad012b7 offset: 0 snap -1 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1771: read oid 49 snap -1 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1771: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1772: read oid 19 snap -1 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1772: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1773: read oid 50 snap -1 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1773: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-13T11:33:40.343 INFO:tasks.rados.rados.0.smithi204.stdout:1774: write oid 15 current snap is 0 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1761: finishing write tid 3 to smithi20438004-30 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1761: finishing write tid 4 to smithi20438004-30 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1761: oid 30 updating version 0 to 88 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1761: oid 30 updating version 88 to 89 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1761: oid 30 version 89 is already newer than 87 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 89 (ObjNum 370 snap 0 seq_num 370) dirty exists 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1761: left oid 30 (ObjNum 370 snap 0 seq_num 370) 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1762: finishing write tid 1 to smithi20438004-9 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1762: finishing write tid 2 to smithi20438004-9 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1762: finishing write tid 3 to smithi20438004-9 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1762: finishing write tid 4 to smithi20438004-9 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1762: oid 9 updating version 0 to 64 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1762: oid 9 updating version 64 to 65 2024-09-13T11:33:40.344 INFO:tasks.rados.rados.0.smithi204.stdout:1762: oid 9 version 65 is already newer than 63 2024-09-13T11:33:40.345 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 65 (ObjNum 371 snap 0 seq_num 371) dirty exists 2024-09-13T11:33:40.345 INFO:tasks.rados.rados.0.smithi204.stdout:1762: left oid 9 (ObjNum 371 snap 0 seq_num 371) 2024-09-13T11:33:40.345 INFO:tasks.rados.rados.0.smithi204.stdout:1763: finishing write tid 1 to smithi20438004-21 2024-09-13T11:33:40.345 INFO:tasks.rados.rados.0.smithi204.stdout:1774: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-13T11:33:40.349 INFO:tasks.rados.rados.0.smithi204.stdout:1774: writing smithi20438004-15 from 669011 to 1238185 tid 1 2024-09-13T11:33:40.356 INFO:tasks.rados.rados.0.smithi204.stdout:1774: writing smithi20438004-15 from 2010607 to 2269011 tid 2 2024-09-13T11:33:40.356 INFO:tasks.rados.rados.0.smithi204.stdout:1761: done (13 left) 2024-09-13T11:33:40.356 INFO:tasks.rados.rados.0.smithi204.stdout:1762: done (12 left) 2024-09-13T11:33:40.356 INFO:tasks.rados.rados.0.smithi204.stdout:1775: read oid 29 snap -1 2024-09-13T11:33:40.356 INFO:tasks.rados.rados.0.smithi204.stdout:1775: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:40.356 INFO:tasks.rados.rados.0.smithi204.stdout:1776: set_chunk oid 22 target oid 2024-09-13T11:33:40.360 INFO:tasks.rados.rados.0.smithi204.stdout:1776: set_chunk oid 22 offset: 2625024 length: 5632 target oid 07fa8a94dd06b17cdd8a23295f9687cd861be80591e8ab912dafabf21117f264 offset: 0 snap -1 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1777: read oid 30 snap -1 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1777: expect (ObjNum 370 snap 0 seq_num 370) 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1778: write oid 26 current snap is 0 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1763: finishing write tid 2 to smithi20438004-21 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1763: finishing write tid 3 to smithi20438004-21 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1763: finishing write tid 4 to smithi20438004-21 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1763: oid 21 updating version 0 to 40 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1763: oid 21 version 40 is already newer than 39 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1763: oid 21 updating version 40 to 41 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 41 (ObjNum 372 snap 0 seq_num 372) dirty exists 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1763: left oid 21 (ObjNum 372 snap 0 seq_num 372) 2024-09-13T11:33:40.361 INFO:tasks.rados.rados.0.smithi204.stdout:1778: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-13T11:33:40.368 INFO:tasks.rados.rados.0.smithi204.stdout:1778: writing smithi20438004-26 from 685818 to 1330241 tid 1 2024-09-13T11:33:40.374 INFO:tasks.rados.rados.0.smithi204.stdout:1778: writing smithi20438004-26 from 2069089 to 2285818 tid 2 2024-09-13T11:33:40.375 INFO:tasks.rados.rados.0.smithi204.stdout:1763: done (15 left) 2024-09-13T11:33:40.375 INFO:tasks.rados.rados.0.smithi204.stdout:1779: read oid 44 snap -1 2024-09-13T11:33:40.375 INFO:tasks.rados.rados.0.smithi204.stdout:1779: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-13T11:33:40.375 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:40.375 INFO:tasks.rados.rados.0.smithi204.stdout:1764: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-13T11:33:40.375 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.333522416999998, 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.427 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.431 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.323247850000001, 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:33:40.435 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.436 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.436 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:40.436 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:40.436 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.310579949000001, 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:40.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.440 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.303970101999994, 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.441 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:33:40.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.295593072000003, 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:33:40.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.270461893000004, 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.450 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:33:40.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.261560614999993, 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:33:40.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.253962892000004, 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.459 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:40.463 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.248558962999994, 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:33:40.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:33:40.468 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.235935272999996, 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:33:40.472 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.221755822999995, 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:33:40.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:33:40.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:33:40.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.213537654999996, 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:33:40.477 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:33:40.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:33:40.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:33:40.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:33:40.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.182714251999997, 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.484 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:40.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:33:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:33:40.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:33:40.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.491 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.035999465000003, 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.492 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:40.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:33:40.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:33:40.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 64.330827412999994, 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:40.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:40.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:33:40.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 63.582875043000001, 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013151014000000001, 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.503 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842045+0000", 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865e-05 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848182+0000", 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061372370000000002 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848205+0000", 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.298e-05 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848218+0000", 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.348e-05 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848300+0000", 2024-09-13T11:33:40.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1766000000000003e-05 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848327+0000", 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6446999999999999e-05 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855119+0000", 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067926490000000004 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855177+0000", 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7590000000000003e-05 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1458 4.e 4:7984cf84:::smithi20438004-46:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 63.397262990999998, 2024-09-13T11:33:40.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012793834, 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1458 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027640+0000", 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.046e-06 2024-09-13T11:33:40.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027651+0000", 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.102e-05 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027669+0000", 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027681+0000", 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2177999999999999e-05 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027762+0000", 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0331e-05 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027868+0000", 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106443 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.040432+0000", 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012563632999999999 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1787 4.e 4:7022fc62:::smithi20438004-6:head [read 188928~1265152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 53.151170299999997, 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013752110999999999, 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1787 2024-09-13T11:33:40.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273731+0000", 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273734+0000", 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3510000000000002e-06 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273741+0000", 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273787+0000", 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6198000000000001e-05 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274432+0000", 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064478300000000001 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274461+0000", 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9051999999999999e-05 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281751+0000", 2024-09-13T11:33:40.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007289963 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281984+0000", 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023257099999999999 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.287483+0000", 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054989829999999998 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.519 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2762 4.e 4:7984cf84:::smithi20438004-46:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e66)", 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 33.528712882000001, 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022335361000000001, 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.520 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2762 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896189+0000", 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3519999999999999e-06 2024-09-13T11:33:40.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896196+0000", 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896210+0000", 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3439e-05 2024-09-13T11:33:40.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896217+0000", 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5889999999999997e-06 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896278+0000", 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1072000000000001e-05 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896338+0000", 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9998999999999999e-05 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.918523+0000", 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022185099999999999 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3763 4.1 4:8533c921:::smithi20438004-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 11.291157881, 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010966191, 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.526 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3763 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:40.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133745+0000", 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.75e-06 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133752+0000", 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133767+0000", 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4577e-05 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133774+0000", 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:40.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133836+0000", 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1787999999999999e-05 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133896+0000", 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9768000000000001e-05 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.144709+0000", 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010813285000000001 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.530 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.531 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.531 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:40.531 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:40.531 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:40.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:33:40.533 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:40.530+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:40.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:40.534 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.534 INFO:tasks.rados.rados.0.smithi204.stdout:1767: expect (ObjNum 368 snap 0 seq_num 368) 2024-09-13T11:33:40.534 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.633 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:40.632+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:40.643 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1320304 2024-09-13T11:33:40.655 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:40.655 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:40.655 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:40.655 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:40.665 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:40.677 INFO:tasks.rados.rados.0.smithi204.stdout:1765: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-13T11:33:40.677 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.734 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:40.733+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:40.774 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:40.774 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:40.774 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:40.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:40.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:40.824 INFO:tasks.rados.rados.0.smithi204.stdout:1766: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-13T11:33:40.825 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:40.869 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:40.867+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:40.892 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:40.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:40.892 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:40.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 95.152771092999998, 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:33:40.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:33:40.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 74.412845465999993, 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.897 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:33:40.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:33:40.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:33:40.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 74.407786908000006, 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:33:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:33:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 74.407363086000004, 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:33:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:33:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:33:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 74.407345042000003, 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:33:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:33:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:33:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.691104269999997, 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:33:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:33:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.677263480999997, 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:33:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:33:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:33:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:33:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.578295507000007, 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:33:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1752 4.9 4:9441befd:::smithi20438004-8:head [write 1527054~766050] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 54.496586569999998, 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062506259999999996, 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1752 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393364+0000", 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.393365+0000", 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0720000000000001e-06 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394108+0000", 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074250399999999995 2024-09-13T11:33:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394120+0000", 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1990999999999999e-05 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394125+0000", 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7270000000000004e-06 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394182+0000", 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6181000000000003e-05 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394248+0000", 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6848999999999996e-05 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.394333+0000", 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4970999999999997e-05 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.396066+0000", 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001732546 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.397394+0000", 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013282680000000001 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399415+0000", 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002021063 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399585+0000", 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016999699999999999 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.399615+0000", 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9456999999999999e-05 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2293 4.9 4:9441befd:::smithi20438004-8:head [write 1586353~512385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.618077792999998, 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066015969999999998, 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2293 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271874+0000", 2024-09-13T11:33:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.497e-06 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272359+0000", 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048419700000000001 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272399+0000", 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0754000000000003e-05 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272410+0000", 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272470+0000", 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0603999999999998e-05 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272575+0000", 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104901 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272731+0000", 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015523300000000001 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.274392+0000", 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016616070000000001 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.276887+0000", 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024952360000000001 2024-09-13T11:33:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278386+0000", 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014986540000000001 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278439+0000", 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2624999999999998e-05 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278475+0000", 2024-09-13T11:33:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5812999999999997e-05 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2494 4.d e63/31 4:b15c1843:::smithi20438004-33:head v 63'65, mlcod=63'62)", 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 40.240864084000002, 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064248049999999996, 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649088+0000", 2024-09-13T11:33:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.78e-07 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649096+0000", 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3790000000000006e-06 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649108+0000", 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1751e-05 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649116+0000", 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3850000000000002e-06 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649307+0000", 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019067199999999999 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649325+0000", 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7859e-05 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655462+0000", 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061369980000000003 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655511+0000", 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9883000000000003e-05 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.4.22:41 3.1 e63/51 3:b93f8350:::9968f6d703223c53b55081382c31f0a72d88ed4f88670426e6b8ab5df371e41b:head v 63'230, mlcod=63'227)", 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 39.788494599000003, 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063303989999999996, 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101456+0000", 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101457+0000", 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101463+0000", 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7529999999999998e-06 2024-09-13T11:33:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101473+0000", 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0154e-05 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101483+0000", 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101542+0000", 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9132999999999998e-05 2024-09-13T11:33:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.101560+0000", 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7156e-05 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107733+0000", 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061735740000000002 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.107787+0000", 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3389e-05 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3068 3.1 e72/51 3:ba0ff2fc:::35d746116e015b742d2355e1d921a21f28a06eaf655b8cfcf0782553e2cf9a5a:head v 72'267, mlcod=70'265)", 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.273341221999999, 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073641619999999996, 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620678+0000", 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040678880000000004 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620694+0000", 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.611e-05 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620705+0000", 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1124999999999999e-05 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620768+0000", 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2626000000000007e-05 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620794+0000", 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6446000000000001e-05 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623920+0000", 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003126298 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623974+0000", 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3375000000000003e-05 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3600 4.9 4:94e25a0d:::smithi20438004-3:head [write 527819~624406,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.738841753999999, 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076023949999999996, 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3600 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151112+0000", 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7070000000000001e-06 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151819+0000", 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00070706700000000003 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151834+0000", 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4944e-05 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151841+0000", 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1760000000000004e-06 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151897+0000", 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5948e-05 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151966+0000", 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9283999999999996e-05 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.152076+0000", 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109633 2024-09-13T11:33:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154405+0000", 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023291950000000001 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154749+0000", 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034383100000000001 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158638+0000", 2024-09-13T11:33:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038894250000000002 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158677+0000", 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9167e-05 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158711+0000", 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4017999999999999e-05 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3987 4.b e81/52 4:da92e835:::smithi20438004-50:head v 81'69, mlcod=76'67)", 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.6804865009999999, 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068559900000000002, 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7100000000000001e-07 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210320+0000", 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085537199999999997 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210332+0000", 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.224e-05 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210343+0000", 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0549e-05 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210424+0000", 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1209000000000001e-05 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210450+0000", 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6673e-05 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216271+0000", 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058204060000000002 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216320+0000", 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9370000000000003e-05 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4013 4.9 4:9441befd:::smithi20438004-8:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e81)", 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1959039669999996, 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017394468999999999, 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4013 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694048+0000", 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.026e-06 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694055+0000", 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9249999999999998e-06 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694065+0000", 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9049999999999992e-06 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694070+0000", 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4419999999999997e-06 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694114+0000", 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4341000000000001e-05 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694170+0000", 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5618999999999997e-05 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.711441+0000", 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017271211000000002 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4055 4.b e82/52 4:d738a69d:::smithi20438004-16:head v 82'73, mlcod=81'71)", 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.285585974, 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066268300000000002, 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8400000000000001e-07 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605461+0000", 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001096461 2024-09-13T11:33:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605475+0000", 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3439e-05 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605482+0000", 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3000000000000004e-06 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605560+0000", 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8305000000000003e-05 2024-09-13T11:33:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605584+0000", 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3723e-05 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.610952+0000", 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053675449999999996 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.610992+0000", 2024-09-13T11:33:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9873000000000003e-05 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4060 4.15 4:a9b0c9bd:::smithi20438004-37:head [write 1366083~511456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2751650970000004, 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073320010000000003, 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4060 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:33:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614787+0000", 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9500000000000001e-07 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615260+0000", 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047298899999999999 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615270+0000", 2024-09-13T11:33:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615276+0000", 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.524e-06 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615325+0000", 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9530000000000002e-05 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615404+0000", 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9208000000000005e-05 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615492+0000", 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7163999999999994e-05 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.617037+0000", 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001545374 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.619988+0000", 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029509470000000002 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622041+0000", 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002052824 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622091+0000", 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:33:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622118+0000", 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7016e-05 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4061 4.15 4:a9b0c9bd:::smithi20438004-37:head [setxattr _header (56) in=7b,truncate 2016906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2746621359999999, 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069574540000000001, 2024-09-13T11:33:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4061 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0200000000000003e-07 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615291+0000", 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:33:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615296+0000", 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9790000000000001e-06 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615299+0000", 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2040000000000002e-06 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616228+0000", 2024-09-13T11:33:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000928213 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616282+0000", 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4147999999999997e-05 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616369+0000", 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7595000000000004e-05 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.617083+0000", 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071378200000000004 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622163+0000", 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050794780000000001 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622189+0000", 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6539000000000001e-05 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622216+0000", 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6443000000000001e-05 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622246+0000", 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0440999999999999e-05 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4062 4.15 4:a9b0c9bd:::smithi20438004-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:33:41.001 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.2746283829999996, 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070487859999999996, 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4062 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:41.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615323+0000", 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:41.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615326+0000", 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615334+0000", 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9300000000000003e-06 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615342+0000", 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4610000000000002e-06 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616576+0000", 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001234787 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616622+0000", 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549e-05 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622271+0000", 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056493400000000001 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622313+0000", 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1616999999999998e-05 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622371+0000", 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8252999999999999e-05 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:41.006 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:41.008 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.008 INFO:tasks.rados.rados.0.smithi204.stdout:1768: finishing set_chunk to oid 43 2024-09-13T11:33:41.008 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 117 (ObjNum 281 snap 0 seq_num 281) dirty exists 2024-09-13T11:33:41.008 INFO:tasks.rados.rados.0.smithi204.stdout:1769: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:41.008 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 136605 2024-09-13T11:33:41.008 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:41.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:41.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:41.010 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:41.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:41.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:41.012 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:41.006+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:41.021 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:41.021 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:41.038 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1183171 2024-09-13T11:33:41.038 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:41.038 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:41.055 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:41.055 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:41.062 INFO:tasks.rados.rados.0.smithi204.stdout:1770: finishing set_chunk to oid 5 2024-09-13T11:33:41.062 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 3256320 length: 203264 2024-09-13T11:33:41.062 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 0 (ObjNum 343 snap 0 seq_num 343) dirty exists 2024-09-13T11:33:41.062 INFO:tasks.rados.rados.0.smithi204.stdout:1771: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-13T11:33:41.073 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:41.074 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:41.091 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:41.091 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:41.102 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1606756 2024-09-13T11:33:41.108 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:41.107+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:41.109 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:41.109 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:41.125 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:41.126 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:41.126 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:41.129 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.130 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:41.130 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:41.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:41.143 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:41.144 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:41.160 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:41.169 INFO:tasks.rados.rados.0.smithi204.stdout:1773: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-13T11:33:41.169 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.177 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:41.178 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:41.181 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:41.223 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:41.221+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:41.250 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:41.250 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:41.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:41.284 INFO:tasks.rados.rados.0.smithi204.stdout:1772: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-13T11:33:41.310 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:41.308+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:41.315 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.357 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.400 INFO:tasks.rados.rados.0.smithi204.stdout:1774: finishing write tid 1 to smithi20438004-15 2024-09-13T11:33:41.400 INFO:tasks.rados.rados.0.smithi204.stdout:1774: finishing write tid 2 to smithi20438004-15 2024-09-13T11:33:41.400 INFO:tasks.rados.rados.0.smithi204.stdout:1774: finishing write tid 3 to smithi20438004-15 2024-09-13T11:33:41.400 INFO:tasks.rados.rados.0.smithi204.stdout:1774: finishing write tid 4 to smithi20438004-15 2024-09-13T11:33:41.400 INFO:tasks.rados.rados.0.smithi204.stdout:1774: oid 15 updating version 0 to 114 2024-09-13T11:33:41.400 INFO:tasks.rados.rados.0.smithi204.stdout:1774: oid 15 updating version 114 to 115 2024-09-13T11:33:41.400 INFO:tasks.rados.rados.0.smithi204.stdout:1774: oid 15 updating version 115 to 116 2024-09-13T11:33:41.401 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 15 v 116 (ObjNum 373 snap 0 seq_num 373) dirty exists 2024-09-13T11:33:41.401 INFO:tasks.rados.rados.0.smithi204.stdout:1774: left oid 15 (ObjNum 373 snap 0 seq_num 373) 2024-09-13T11:33:41.401 INFO:tasks.rados.rados.0.smithi204.stdout:1776: finishing set_chunk to oid 22 2024-09-13T11:33:41.401 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 109 (ObjNum 352 snap 0 seq_num 352) dirty exists 2024-09-13T11:33:41.401 INFO:tasks.rados.rados.0.smithi204.stdout:1777: expect (ObjNum 370 snap 0 seq_num 370) 2024-09-13T11:33:41.406 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:41.407 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:41.411 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:41.410+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:41.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:41.460 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1263527 2024-09-13T11:33:41.466 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:33:41.512 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:41.511+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:41.524 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:41.524 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:41.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:41.539 INFO:tasks.rados.rados.0.smithi204.stdout:1775: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:41.539 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.612 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:41.610+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:41.639 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:41.640 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:41.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:41.663 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.714 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:41.712+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:41.728 INFO:tasks.rados.rados.0.smithi204.stdout:1778: finishing write tid 1 to smithi20438004-26 2024-09-13T11:33:41.728 INFO:tasks.rados.rados.0.smithi204.stdout:1778: finishing write tid 2 to smithi20438004-26 2024-09-13T11:33:41.728 INFO:tasks.rados.rados.0.smithi204.stdout:1778: finishing write tid 3 to smithi20438004-26 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1778: finishing write tid 4 to smithi20438004-26 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1778: oid 26 updating version 0 to 72 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1778: oid 26 updating version 72 to 74 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1778: oid 26 version 74 is already newer than 73 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 74 (ObjNum 374 snap 0 seq_num 374) dirty exists 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1778: left oid 26 (ObjNum 374 snap 0 seq_num 374) 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1764: done (15 left) 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1765: done (14 left) 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1766: done (13 left) 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1767: done (12 left) 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1768: done (11 left) 2024-09-13T11:33:41.729 INFO:tasks.rados.rados.0.smithi204.stdout:1769: done (10 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1770: done (9 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1771: done (8 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1772: done (7 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1773: done (6 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1774: done (5 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1775: done (4 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1776: done (3 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1777: done (2 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1778: done (1 left) 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1780: write (excl) oid 14 current snap is 0 2024-09-13T11:33:41.730 INFO:tasks.rados.rados.0.smithi204.stdout:1779: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-13T11:33:41.771 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:41.796 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:41.796 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:41.802 INFO:tasks.rados.rados.0.smithi204.stdout:1780: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-13T11:33:41.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:41.810 INFO:tasks.rados.rados.0.smithi204.stdout:1780: writing smithi20438004-14 from 702625 to 1422297 tid 1 2024-09-13T11:33:41.815 INFO:tasks.rados.rados.0.smithi204.stdout:1780: writing smithi20438004-14 from 2127571 to 2302625 tid 2 2024-09-13T11:33:41.815 INFO:tasks.rados.rados.0.smithi204.stdout:1779: done (1 left) 2024-09-13T11:33:41.816 INFO:tasks.rados.rados.0.smithi204.stdout:1781: write oid 39 current snap is 0 2024-09-13T11:33:41.816 INFO:tasks.rados.rados.0.smithi204.stdout:1781: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-13T11:33:41.823 INFO:tasks.rados.rados.0.smithi204.stdout:1781: writing smithi20438004-39 from 719432 to 1514353 tid 1 2024-09-13T11:33:41.827 INFO:tasks.rados.rados.0.smithi204.stdout:1781: writing smithi20438004-39 from 2186053 to 2319432 tid 2 2024-09-13T11:33:41.827 INFO:tasks.rados.rados.0.smithi204.stdout:1782: read oid 32 snap -1 2024-09-13T11:33:41.827 INFO:tasks.rados.rados.0.smithi204.stdout:1782: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:41.828 INFO:tasks.rados.rados.0.smithi204.stdout:1783: set_chunk oid 7 target oid 2024-09-13T11:33:41.837 INFO:tasks.rados.rados.0.smithi204.stdout:1783: set_chunk oid 7 offset: 2266624 length: 675328 target oid a5546976bb72d92a9906380f2ebdcf64b9b17bf8a210f520eda06bf255ad61a3 offset: 0 snap -1 2024-09-13T11:33:41.837 INFO:tasks.rados.rados.0.smithi204.stdout:1784: read oid 4 snap -1 2024-09-13T11:33:41.837 INFO:tasks.rados.rados.0.smithi204.stdout:1784: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-13T11:33:41.838 INFO:tasks.rados.rados.0.smithi204.stdout:1785: write (excl) oid 25 current snap is 0 2024-09-13T11:33:41.838 INFO:tasks.rados.rados.0.smithi204.stdout:1780: finishing write tid 1 to smithi20438004-14 2024-09-13T11:33:41.838 INFO:tasks.rados.rados.0.smithi204.stdout:1785: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-13T11:33:41.843 INFO:tasks.rados.rados.0.smithi204.stdout:1785: writing smithi20438004-25 from 736239 to 1206409 tid 1 2024-09-13T11:33:41.849 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:41.848+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:41.850 INFO:tasks.rados.rados.0.smithi204.stdout:1785: writing smithi20438004-25 from 1728182 to 2299964 tid 2 2024-09-13T11:33:41.850 INFO:tasks.rados.rados.0.smithi204.stdout:1786: tier_promote oid 37 2024-09-13T11:33:41.850 INFO:tasks.rados.rados.0.smithi204.stdout:1780: finishing write tid 2 to smithi20438004-14 2024-09-13T11:33:41.850 INFO:tasks.rados.rados.0.smithi204.stdout:1780: finishing write tid 3 to smithi20438004-14 2024-09-13T11:33:41.851 INFO:tasks.rados.rados.0.smithi204.stdout:1780: finishing write tid 4 to smithi20438004-14 2024-09-13T11:33:41.851 INFO:tasks.rados.rados.0.smithi204.stdout:1780: oid 14 updating version 0 to 132 2024-09-13T11:33:41.851 INFO:tasks.rados.rados.0.smithi204.stdout:1780: oid 14 updating version 132 to 133 2024-09-13T11:33:41.851 INFO:tasks.rados.rados.0.smithi204.stdout:1780: oid 14 updating version 133 to 134 2024-09-13T11:33:41.851 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 134 (ObjNum 375 snap 0 seq_num 375) dirty exists 2024-09-13T11:33:41.851 INFO:tasks.rados.rados.0.smithi204.stdout:1780: left oid 14 (ObjNum 375 snap 0 seq_num 375) 2024-09-13T11:33:41.851 INFO:tasks.rados.rados.0.smithi204.stdout:1782: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:41.893 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1866513 2024-09-13T11:33:41.913 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1647457 2024-09-13T11:33:41.915 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:41.914+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:41.926 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:41.926 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:41.927 INFO:tasks.rados.rados.0.smithi204.stdout:1781: finishing write tid 1 to smithi20438004-39 2024-09-13T11:33:41.927 INFO:tasks.rados.rados.0.smithi204.stdout:1781: finishing write tid 2 to smithi20438004-39 2024-09-13T11:33:41.927 INFO:tasks.rados.rados.0.smithi204.stdout:1781: finishing write tid 3 to smithi20438004-39 2024-09-13T11:33:41.927 INFO:tasks.rados.rados.0.smithi204.stdout:1780: done (6 left) 2024-09-13T11:33:41.927 INFO:tasks.rados.rados.0.smithi204.stdout:1782: done (5 left) 2024-09-13T11:33:41.927 INFO:tasks.rados.rados.0.smithi204.stdout:1787: set_chunk oid 31 target oid 2024-09-13T11:33:41.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:41.937 INFO:tasks.rados.rados.0.smithi204.stdout:1787: set_chunk oid 31 offset: 677888 length: 957440 target oid 7dbf5634ed3591f5e8b0f06a21f85b16b53575aa10e6900651fdfe90e3b2ea7c offset: 0 snap -1 2024-09-13T11:33:41.937 INFO:tasks.rados.rados.0.smithi204.stdout:1788: read oid 45 snap -1 2024-09-13T11:33:41.937 INFO:tasks.rados.rados.0.smithi204.stdout:1788: expect (ObjNum 323 snap 0 seq_num 323) 2024-09-13T11:33:41.937 INFO:tasks.rados.rados.0.smithi204.stdout:1789: read oid 8 snap -1 2024-09-13T11:33:41.937 INFO:tasks.rados.rados.0.smithi204.stdout:1789: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:1790: write oid 28 current snap is 0 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:1781: finishing write tid 4 to smithi20438004-39 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:1781: oid 39 updating version 0 to 126 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:1781: oid 39 updating version 126 to 127 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:1781: oid 39 version 127 is already newer than 125 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 127 (ObjNum 376 snap 0 seq_num 376) dirty exists 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:1781: left oid 39 (ObjNum 376 snap 0 seq_num 376) 2024-09-13T11:33:41.938 INFO:tasks.rados.rados.0.smithi204.stdout:1790: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-13T11:33:41.944 INFO:tasks.rados.rados.0.smithi204.stdout:1790: writing smithi20438004-28 from 753046 to 1298465 tid 1 2024-09-13T11:33:41.949 INFO:tasks.rados.rados.0.smithi204.stdout:1790: writing smithi20438004-28 from 1786664 to 2218457 tid 2 2024-09-13T11:33:41.949 INFO:tasks.rados.rados.0.smithi204.stdout:1781: done (8 left) 2024-09-13T11:33:41.949 INFO:tasks.rados.rados.0.smithi204.stdout:1791: tier_evict oid 38 2024-09-13T11:33:41.949 INFO:tasks.rados.rados.0.smithi204.stdout:1791: tier_evict oid 38 snap -1 2024-09-13T11:33:41.950 INFO:tasks.rados.rados.0.smithi204.stdout:1783: finishing set_chunk to oid 7 2024-09-13T11:33:41.950 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 83 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-13T11:33:41.950 INFO:tasks.rados.rados.0.smithi204.stdout:1784: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-13T11:33:41.967 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1345565 2024-09-13T11:33:41.991 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:42.020 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:42.019+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:42.082 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:42.082 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:42.089 INFO:tasks.rados.rados.0.smithi204.stdout:1785: finishing write tid 1 to smithi20438004-25 2024-09-13T11:33:42.089 INFO:tasks.rados.rados.0.smithi204.stdout:1785: finishing write tid 2 to smithi20438004-25 2024-09-13T11:33:42.089 INFO:tasks.rados.rados.0.smithi204.stdout:1785: finishing write tid 3 to smithi20438004-25 2024-09-13T11:33:42.089 INFO:tasks.rados.rados.0.smithi204.stdout:1785: finishing write tid 4 to smithi20438004-25 2024-09-13T11:33:42.089 INFO:tasks.rados.rados.0.smithi204.stdout:1785: oid 25 updating version 0 to 128 2024-09-13T11:33:42.089 INFO:tasks.rados.rados.0.smithi204.stdout:1785: oid 25 updating version 128 to 129 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1785: oid 25 updating version 129 to 130 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 130 (ObjNum 377 snap 0 seq_num 377) dirty exists 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1785: left oid 25 (ObjNum 377 snap 0 seq_num 377) 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1786: got (0) Success 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 49 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1783: done (8 left) 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1784: done (7 left) 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1785: done (6 left) 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1786: done (5 left) 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1792: write oid 18 current snap is 0 2024-09-13T11:33:42.090 INFO:tasks.rados.rados.0.smithi204.stdout:1792: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-13T11:33:42.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:42.096 INFO:tasks.rados.rados.0.smithi204.stdout:1792: writing smithi20438004-18 from 769853 to 1390521 tid 1 2024-09-13T11:33:42.101 INFO:tasks.rados.rados.0.smithi204.stdout:1792: writing smithi20438004-18 from 1845146 to 2369853 tid 2 2024-09-13T11:33:42.102 INFO:tasks.rados.rados.0.smithi204.stdout:1793: set_chunk oid 12 target oid 2024-09-13T11:33:42.116 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:42.115+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:42.120 INFO:tasks.rados.rados.0.smithi204.stdout:1793: set_chunk oid 12 offset: 627712 length: 1783808 target oid ed1bb18194d877aebe6e9ec49351d9b1d9f2b58bc225b2fdfc115d9c01fbb718 offset: 0 snap -1 2024-09-13T11:33:42.120 INFO:tasks.rados.rados.0.smithi204.stdout:1794: read oid 16 snap -1 2024-09-13T11:33:42.120 INFO:tasks.rados.rados.0.smithi204.stdout:1794: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-13T11:33:42.120 INFO:tasks.rados.rados.0.smithi204.stdout:1795: set_chunk oid 1 target oid 2024-09-13T11:33:42.125 INFO:tasks.rados.rados.0.smithi204.stdout:1795: set_chunk oid 1 offset: 2104832 length: 598016 target oid 43fdf438f81791db2309fa260b51eb7045249776a21626d9f69c1ff241c37993 offset: 0 snap -1 2024-09-13T11:33:42.125 INFO:tasks.rados.rados.0.smithi204.stdout:1796: write (excl) oid 4 current snap is 0 2024-09-13T11:33:42.125 INFO:tasks.rados.rados.0.smithi204.stdout:1796: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-13T11:33:42.133 INFO:tasks.rados.rados.0.smithi204.stdout:1796: writing smithi20438004-4 from 786660 to 1482577 tid 1 2024-09-13T11:33:42.138 INFO:tasks.rados.rados.0.smithi204.stdout:1796: writing smithi20438004-4 from 1903628 to 2339090 tid 2 2024-09-13T11:33:42.138 INFO:tasks.rados.rados.0.smithi204.stdout:1797: write (excl) oid 6 current snap is 0 2024-09-13T11:33:42.138 INFO:tasks.rados.rados.0.smithi204.stdout:1797: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-13T11:33:42.143 INFO:tasks.rados.rados.0.smithi204.stdout:1797: writing smithi20438004-6 from 403467 to 890986 tid 1 2024-09-13T11:33:42.150 INFO:tasks.rados.rados.0.smithi204.stdout:1797: writing smithi20438004-6 from 1562110 to 2141230 tid 2 2024-09-13T11:33:42.150 INFO:tasks.rados.rados.0.smithi204.stdout:1798: set_chunk oid 27 target oid 2024-09-13T11:33:42.162 INFO:tasks.rados.rados.0.smithi204.stdout:1798: set_chunk oid 27 offset: 211456 length: 950272 target oid 130d41c184dcbbb76d464d336fc7708883ab17e8920677c6041054988573dded offset: 0 snap -1 2024-09-13T11:33:42.162 INFO:tasks.rados.rados.0.smithi204.stdout:1799: set_chunk oid 14 target oid 2024-09-13T11:33:42.162 INFO:tasks.rados.rados.0.smithi204.stdout:1787: finishing set_chunk to oid 31 2024-09-13T11:33:42.162 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 62 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-13T11:33:42.165 INFO:tasks.rados.rados.0.smithi204.stdout:1799: set_chunk oid 14 offset: 234496 length: 336896 target oid 6da098bcbe04c52044c6e7dd3c1f8730ee6179b6152ff6e6dffb2c6b3e2f0fc2 offset: 0 snap -1 2024-09-13T11:33:42.165 INFO:tasks.rados.rados.0.smithi204.stdout:1787: done (12 left) 2024-09-13T11:33:42.165 INFO:tasks.rados.rados.0.smithi204.stdout:1800: set_chunk oid 35 target oid 2024-09-13T11:33:42.175 INFO:tasks.rados.rados.0.smithi204.stdout:1800: set_chunk oid 35 offset: 1587712 length: 1114624 target oid f6ccb87ac7faedafaf641154fd342d6f919768051142ae54fb5340524f51b241 offset: 0 snap -1 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout:1801: read oid 20 snap -1 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout:1801: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout:1802: read oid 36 snap -1 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout:1802: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout:1803: read oid 50 snap -1 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout:1803: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:42.176 INFO:tasks.rados.rados.0.smithi204.stdout:1789: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-13T11:33:42.217 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:42.215+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:42.218 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2253292 2024-09-13T11:33:42.233 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:42.233 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:42.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:42.318 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:42.316+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:42.331 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 707296 2024-09-13T11:33:42.344 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:42.344 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:42.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:42.403 INFO:tasks.rados.rados.0.smithi204.stdout:1788: expect (ObjNum 323 snap 0 seq_num 323) 2024-09-13T11:33:42.418 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:42.417+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:42.498 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:42.498 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:42.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:42.534 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2255684 2024-09-13T11:33:42.550 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:42.550 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:42.549+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:42.610 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:42.610 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:42.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:42.620 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:42.618+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1790: finishing write tid 1 to smithi20438004-28 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1791: got (22) Invalid argument 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1790: finishing write tid 2 to smithi20438004-28 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1790: finishing write tid 3 to smithi20438004-28 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1788: done (15 left) 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1789: done (14 left) 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1791: done (13 left) 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1804: read oid 2 snap -1 2024-09-13T11:33:42.630 INFO:tasks.rados.rados.0.smithi204.stdout:1804: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1790: finishing write tid 4 to smithi20438004-28 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1790: oid 28 updating version 0 to 87 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1790: oid 28 updating version 87 to 88 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1790: oid 28 version 88 is already newer than 86 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 88 (ObjNum 378 snap 0 seq_num 378) dirty exists 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1790: left oid 28 (ObjNum 378 snap 0 seq_num 378) 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1792: finishing write tid 1 to smithi20438004-18 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1792: finishing write tid 2 to smithi20438004-18 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1790: done (13 left) 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1805: read oid 33 snap -1 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1805: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1806: write oid 49 current snap is 0 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1792: finishing write tid 3 to smithi20438004-18 2024-09-13T11:33:42.631 INFO:tasks.rados.rados.0.smithi204.stdout:1806: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-13T11:33:42.635 INFO:tasks.rados.rados.0.smithi204.stdout:1806: writing smithi20438004-49 from 420274 to 983042 tid 1 2024-09-13T11:33:42.642 INFO:tasks.rados.rados.0.smithi204.stdout:1806: writing smithi20438004-49 from 1620592 to 2059723 tid 2 2024-09-13T11:33:42.642 INFO:tasks.rados.rados.0.smithi204.stdout:1807: set_chunk oid 47 target oid 2024-09-13T11:33:42.644 INFO:tasks.rados.rados.0.smithi204.stdout:1807: set_chunk oid 47 offset: 2376704 length: 55296 target oid 35295da1d5eca0b6db3168c0a64a2d61ed3ae8ca283dd4aa6116aa04703dcb60 offset: 0 snap -1 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1792: finishing write tid 4 to smithi20438004-18 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1792: oid 18 updating version 0 to 145 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1792: oid 18 version 145 is already newer than 144 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1792: oid 18 updating version 145 to 146 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 146 (ObjNum 379 snap 0 seq_num 379) dirty exists 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1792: left oid 18 (ObjNum 379 snap 0 seq_num 379) 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1792: done (15 left) 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1808: read oid 38 snap -1 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1808: expect (ObjNum 351 snap 0 seq_num 351) 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1793: finishing set_chunk to oid 12 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 86 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-09-13T11:33:42.645 INFO:tasks.rados.rados.0.smithi204.stdout:1794: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-13T11:33:42.687 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:42.721 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:42.719+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:42.761 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:42.761 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:42.770 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1799366 2024-09-13T11:33:42.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:1795: finishing set_chunk to oid 1 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 1 set_chunk returned error code -95 offset: 2104832 length: 598016 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 0 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:1796: finishing write tid 1 to smithi20438004-4 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:1796: finishing write tid 2 to smithi20438004-4 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:1796: finishing write tid 3 to smithi20438004-4 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:1796: finishing write tid 4 to smithi20438004-4 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:1796: oid 4 updating version 0 to 91 2024-09-13T11:33:42.773 INFO:tasks.rados.rados.0.smithi204.stdout:1796: oid 4 version 91 is already newer than 90 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1796: oid 4 updating version 91 to 92 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 92 (ObjNum 380 snap 0 seq_num 380) dirty exists 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1796: left oid 4 (ObjNum 380 snap 0 seq_num 380) 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1797: finishing write tid 1 to smithi20438004-6 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1797: finishing write tid 2 to smithi20438004-6 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1797: finishing write tid 3 to smithi20438004-6 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1793: done (15 left) 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1794: done (14 left) 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1795: done (13 left) 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1796: done (12 left) 2024-09-13T11:33:42.774 INFO:tasks.rados.rados.0.smithi204.stdout:1809: write (excl) oid 8 current snap is 0 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:1797: finishing write tid 4 to smithi20438004-6 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:1797: oid 6 updating version 0 to 102 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:1797: oid 6 version 102 is already newer than 101 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:1797: oid 6 updating version 102 to 103 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 103 (ObjNum 381 snap 0 seq_num 381) dirty exists 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:1797: left oid 6 (ObjNum 381 snap 0 seq_num 381) 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:1798: finishing set_chunk to oid 27 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 121 (ObjNum 360 snap 0 seq_num 360) dirty exists 2024-09-13T11:33:42.775 INFO:tasks.rados.rados.0.smithi204.stdout:1799: finishing set_chunk to oid 14 2024-09-13T11:33:42.776 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 134 (ObjNum 375 snap 0 seq_num 375) dirty exists 2024-09-13T11:33:42.776 INFO:tasks.rados.rados.0.smithi204.stdout:1800: finishing set_chunk to oid 35 2024-09-13T11:33:42.776 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 1587712 length: 1114624 2024-09-13T11:33:42.776 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-09-13T11:33:42.776 INFO:tasks.rados.rados.0.smithi204.stdout:1803: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-13T11:33:42.821 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:42.819+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:42.851 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3035157 2024-09-13T11:33:42.860 INFO:tasks.rados.rados.0.smithi204.stdout:1802: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-13T11:33:42.914 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:42.914 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:42.914 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3261127 2024-09-13T11:33:42.921 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:42.920+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:42.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:42.933 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1023325 2024-09-13T11:33:43.023 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:43.022+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:43.026 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:43.026 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:43.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:43.068 INFO:tasks.rados.rados.0.smithi204.stdout:1801: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-13T11:33:43.099 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 665979 2024-09-13T11:33:43.124 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:43.122+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:43.137 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:43.138 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:43.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:43.158 INFO:tasks.rados.rados.0.smithi204.stdout:1804: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-13T11:33:43.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:43.225 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:43.224+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:43.249 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:43.249 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:43.257 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:43.307 INFO:tasks.rados.rados.0.smithi204.stdout:1806: finishing write tid 1 to smithi20438004-49 2024-09-13T11:33:43.307 INFO:tasks.rados.rados.0.smithi204.stdout:1805: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-13T11:33:43.307 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:43.320 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:43.321 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:43.326 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:43.324+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:43.339 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:43.339 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:43.339 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:43.356 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:43.357 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:43.360 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:43.361 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:43.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:43.373 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:43.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:43.390 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:43.390 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:43.407 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:43.408 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:43.424 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:43.425 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:43.427 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:43.425+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:43.433 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3411235 2024-09-13T11:33:43.440 INFO:tasks.rados.rados.0.smithi204.stdout:1806: finishing write tid 2 to smithi20438004-49 2024-09-13T11:33:43.440 INFO:tasks.rados.rados.0.smithi204.stdout:1807: finishing set_chunk to oid 47 2024-09-13T11:33:43.440 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 2376704 length: 55296 2024-09-13T11:33:43.440 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-09-13T11:33:43.440 INFO:tasks.rados.rados.0.smithi204.stdout:1806: finishing write tid 3 to smithi20438004-49 2024-09-13T11:33:43.440 INFO:tasks.rados.rados.0.smithi204.stdout:1809: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-13T11:33:43.441 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:43.441 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:43.445 INFO:tasks.rados.rados.0.smithi204.stdout:1809: writing smithi20438004-8 from 437081 to 1075098 tid 1 2024-09-13T11:33:43.452 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:43.452 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:43.452 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:43.453 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-13T11:33:43.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1809: writing smithi20438004-8 from 1679074 to 2261863 tid 2 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1797: done (12 left) 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1798: done (11 left) 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1799: done (10 left) 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1800: done (9 left) 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1801: done (8 left) 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1802: done (7 left) 2024-09-13T11:33:43.455 INFO:tasks.rados.rados.0.smithi204.stdout:1803: done (6 left) 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1804: done (5 left) 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1805: done (4 left) 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1807: done (3 left) 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1810: read oid 13 snap -1 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1810: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1811: read oid 29 snap -1 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1811: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1812: write oid 10 current snap is 0 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1806: finishing write tid 4 to smithi20438004-49 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1806: oid 49 updating version 0 to 43 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1806: oid 49 version 43 is already newer than 42 2024-09-13T11:33:43.456 INFO:tasks.rados.rados.0.smithi204.stdout:1806: oid 49 updating version 43 to 44 2024-09-13T11:33:43.457 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 44 (ObjNum 382 snap 0 seq_num 382) dirty exists 2024-09-13T11:33:43.457 INFO:tasks.rados.rados.0.smithi204.stdout:1806: left oid 49 (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:43.457 INFO:tasks.rados.rados.0.smithi204.stdout:1812: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-13T11:33:43.459 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:43.459 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:43.460 INFO:tasks.rados.rados.0.smithi204.stdout:1812: writing smithi20438004-10 from 453888 to 1167154 tid 1 2024-09-13T11:33:43.466 INFO:tasks.rados.rados.0.smithi204.stdout:1812: writing smithi20438004-10 from 1737556 to 2180356 tid 2 2024-09-13T11:33:43.466 INFO:tasks.rados.rados.0.smithi204.stdout:1806: done (5 left) 2024-09-13T11:33:43.467 INFO:tasks.rados.rados.0.smithi204.stdout:1813: tier_evict oid 36 2024-09-13T11:33:43.467 INFO:tasks.rados.rados.0.smithi204.stdout:1813: tier_evict oid 36 snap -1 2024-09-13T11:33:43.467 INFO:tasks.rados.rados.0.smithi204.stdout:1808: expect (ObjNum 351 snap 0 seq_num 351) 2024-09-13T11:33:43.478 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:43.478 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:43.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:33:43.490 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:43.490 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:43.495 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:43.496 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:43.496 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:43.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:43.516 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:43.517 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:43.528 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:43.527+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:43.533 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:43.534 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:43.552 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:43.552 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:43.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:43.604 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:43.604 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:43.605 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:43.605 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:43.605 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:43.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:43.657 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:43.656+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:43.720 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:201 3.1 3:b265d015:::smithi20438004-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.632943194000006, 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030631999, 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 201 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084693+0000", 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4739999999999999e-06 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.884e-06 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2858e-05 2024-09-13T11:33:43.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084748+0000", 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3113e-05 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084807+0000", 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8523999999999997e-05 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084930+0000", 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012317099999999999 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097750+0000", 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012820575000000001 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107462+0000", 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097120990000000001 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114842+0000", 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073796520000000004 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115192+0000", 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035007400000000002 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115249+0000", 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7277000000000001e-05 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115323+0000", 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3298000000000002e-05 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:202 3.1 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.632861747000007, 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055756029999999998, 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.727 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084773+0000", 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084777+0000", 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3540000000000002e-06 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084784+0000", 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084789+0000", 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5480000000000001e-06 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085081+0000", 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000291673 2024-09-13T11:33:43.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085197+0000", 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011550800000000001 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109819+0000", 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024622326999999999 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115554+0000", 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057346100000000002 2024-09-13T11:33:43.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.139296+0000", 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023742025999999999 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140477+0000", 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011813660000000001 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140507+0000", 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0307e-05 2024-09-13T11:33:43.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140528+0000", 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0934e-05 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:203 3.1 3:970ba768:::smithi20438004-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.632818979000007, 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055588820999999997, 2024-09-13T11:33:43.733 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1900000000000003e-07 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084824+0000", 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3110000000000006e-06 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084831+0000", 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4869999999999996e-06 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084839+0000", 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7840000000000004e-06 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085318+0000", 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047915700000000002 2024-09-13T11:33:43.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085404+0000", 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5576999999999997e-05 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.108042+0000", 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022638703999999999 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115550+0000", 2024-09-13T11:33:43.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075072150000000002 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132897+0000", 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017347621000000001 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140260+0000", 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073629139999999999 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140331+0000", 2024-09-13T11:33:43.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0994999999999994e-05 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140404+0000", 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2537e-05 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:206 3.1 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.632783357999998, 2024-09-13T11:33:43.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022595217000000001, 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1799999999999995e-07 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084854+0000", 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1650000000000002e-06 2024-09-13T11:33:43.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084861+0000", 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1300000000000003e-06 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084868+0000", 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.55e-06 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085510+0000", 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064168099999999996 2024-09-13T11:33:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085595+0000", 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5643999999999995e-05 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.091806+0000", 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062107220000000001 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107236+0000", 2024-09-13T11:33:43.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015429897999999999 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107281+0000", 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4994000000000003e-05 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107302+0000", 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1163e-05 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107375+0000", 2024-09-13T11:33:43.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2915000000000002e-05 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107446+0000", 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0837e-05 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:210 3.1 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.745 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.632756223000001, 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047980961000000003, 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 210 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084883+0000", 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084890+0000", 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0650000000000001e-06 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084896+0000", 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9000000000000003e-06 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085703+0000", 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000807097 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085822+0000", 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119098 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.104833+0000", 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019011389 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115543+0000", 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01070981 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127458+0000", 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011915189 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132784+0000", 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053251139999999997 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132825+0000", 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1329999999999999e-05 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132859+0000", 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.751 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:215 3.1 3:84f0de12:::smithi20438004-16 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.632729905999994, 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034172806999999999, 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 215 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0400000000000002e-07 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084907+0000", 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.666e-06 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084915+0000", 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9330000000000001e-06 2024-09-13T11:33:43.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084920+0000", 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8289999999999998e-06 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085942+0000", 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001022029 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.086010+0000", 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8007000000000001e-05 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.100909+0000", 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014898991 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107467+0000", 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065583270000000001 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.117910+0000", 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010443309 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.118971+0000", 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010605510000000001 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119023+0000", 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2114000000000003e-05 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119077+0000", 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3647000000000003e-05 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:218 3.1 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.607047358000003, 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036660768000000003, 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 218 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.759 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110588+0000", 2024-09-13T11:33:43.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.761e-06 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110603+0000", 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4542000000000001e-05 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110616+0000", 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2795e-05 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110628+0000", 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1959e-05 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114882+0000", 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042543379999999999 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000143676 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.120883+0000", 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058572549999999996 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.123587+0000", 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027042709999999998 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144002+0000", 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020414505999999999 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147169+0000", 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031675470000000002 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147204+0000", 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4523999999999997e-05 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147247+0000", 2024-09-13T11:33:43.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3594000000000003e-05 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:219 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.606583001999994, 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.054051957999999997, 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.765 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 219 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111052+0000", 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.093e-06 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111059+0000", 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9410000000000001e-06 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111069+0000", 2024-09-13T11:33:43.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0519999999999999e-05 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111077+0000", 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8290000000000008e-06 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115373+0000", 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042952490000000001 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115459+0000", 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6367999999999997e-05 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.129610+0000", 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014151285 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.135388+0000", 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057775270000000002 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149276+0000", 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013888773 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165015+0000", 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015738682 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165065+0000", 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0170000000000002e-05 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165103+0000", 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7521000000000002e-05 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:221 3.1 3:baf08732:::smithi20438004-22 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.601190231000004, 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034943881000000003, 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.771 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 221 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116445+0000", 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1680000000000001e-06 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116451+0000", 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0379999999999996e-06 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116461+0000", 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0604e-05 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116469+0000", 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8359999999999992e-06 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116516+0000", 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6332999999999998e-05 2024-09-13T11:33:43.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116579+0000", 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3825999999999995e-05 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127789+0000", 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011209869000000001 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132875+0000", 2024-09-13T11:33:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050855780000000003 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149082+0000", 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016207213000000002 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151312+0000", 2024-09-13T11:33:43.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002230358 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151344+0000", 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1081000000000003e-05 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151388+0000", 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3977000000000003e-05 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:230 3.1 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.571039757999998, 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046965185999999999, 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.778 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146596+0000", 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.717e-06 2024-09-13T11:33:43.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146606+0000", 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146620+0000", 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4343e-05 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146630+0000", 2024-09-13T11:33:43.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3989999999999993e-06 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146705+0000", 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5135999999999993e-05 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146828+0000", 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122714 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165695+0000", 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018866951 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.171485+0000", 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057905220000000002 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185089+0000", 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013604132999999999 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193461+0000", 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083717840000000002 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193507+0000", 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5652000000000003e-05 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193559+0000", 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2624000000000003e-05 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.783 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:231 3.1 3:ab075b6a:::smithi20438004-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.570830407000003, 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033558088, 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 231 2024-09-13T11:33:43.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146804+0000", 2024-09-13T11:33:43.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3300000000000002e-07 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146808+0000", 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4790000000000001e-06 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146814+0000", 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3230000000000003e-06 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146817+0000", 2024-09-13T11:33:43.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4769999999999999e-06 2024-09-13T11:33:43.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146955+0000", 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013720299999999999 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147052+0000", 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7231000000000005e-05 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.156769+0000", 2024-09-13T11:33:43.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097173290000000002 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165122+0000", 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083524589999999996 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.173409+0000", 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082871379999999994 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180256+0000", 2024-09-13T11:33:43.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006846871 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180302+0000", 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6545999999999998e-05 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180362+0000", 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9499000000000001e-05 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:235 3.1 3:be4a0686:::smithi20438004-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.567831980999998, 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035212407000000001, 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 235 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.791 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149803+0000", 2024-09-13T11:33:43.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0580000000000001e-06 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149810+0000", 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149820+0000", 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0464e-05 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149826+0000", 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149863+0000", 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7416999999999998e-05 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149983+0000", 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011927800000000001 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.160853+0000", 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010870216 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.167925+0000", 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070717159999999996 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180224+0000", 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012298972 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184892+0000", 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046685839999999999 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184959+0000", 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7105000000000002e-05 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185014+0000", 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5099000000000002e-05 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.796 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:236 3.1 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.567718065999998, 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059325392999999997, 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 236 2024-09-13T11:33:43.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:43.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6599999999999996e-07 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149920+0000", 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149929+0000", 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3769999999999996e-06 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149937+0000", 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150123+0000", 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018567200000000001 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150254+0000", 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130426 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181070+0000", 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030816856 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.190793+0000", 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097220870000000008 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197985+0000", 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007192431 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209174+0000", 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011189536999999999 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209209+0000", 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4359000000000001e-05 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209241+0000", 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2369000000000001e-05 2024-09-13T11:33:43.802 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:237 3.1 3:9a2730a7:::smithi20438004-38 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.565765279000004, 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052738543999999998, 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.803 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 237 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151870+0000", 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151875+0000", 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2859999999999999e-06 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151890+0000", 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5557999999999999e-05 2024-09-13T11:33:43.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151898+0000", 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7500000000000003e-06 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151934+0000", 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5988000000000002e-05 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151999+0000", 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4394999999999996e-05 2024-09-13T11:33:43.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.177811+0000", 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025812831000000001 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185336+0000", 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075243209999999996 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197387+0000", 2024-09-13T11:33:43.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012051345999999999 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204491+0000", 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071034999999999996 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204551+0000", 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0003000000000001e-05 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204607+0000", 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:245 3.1 3:adf25eb4:::smithi20438004-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.536576230999998, 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.03651658, 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 245 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181059+0000", 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181066+0000", 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5119999999999997e-06 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181078+0000", 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2333e-05 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181085+0000", 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4449999999999999e-06 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185114+0000", 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040285579999999998 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1935000000000007e-05 2024-09-13T11:33:43.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.195343+0000", 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010137147000000001 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200849+0000", 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005505874 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.203835+0000", 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002985705 2024-09-13T11:33:43.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217494+0000", 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013658926 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217538+0000", 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4048999999999998e-05 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217574+0000", 2024-09-13T11:33:43.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6618000000000001e-05 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:249 3.1 3:81faf980:::smithi20438004-50 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.523657581999998, 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038339094999999997, 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:43.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 249 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193977+0000", 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.11e-06 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193984+0000", 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0794e-05 2024-09-13T11:33:43.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194002+0000", 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.374e-06 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194061+0000", 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9485999999999999e-05 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194291+0000", 2024-09-13T11:33:43.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000229321 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222511+0000", 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028219767999999999 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.227779+0000", 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052686319999999997 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.231941+0000", 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041617809999999998 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232226+0000", 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028545799999999999 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232277+0000", 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0445000000000002e-05 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232315+0000", 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584999999999997e-05 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:44 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.449233853999999, 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019379767999999999, 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.821 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268402+0000", 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268410+0000", 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2069999999999995e-06 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268426+0000", 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5588999999999999e-05 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268437+0000", 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0773e-05 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270340+0000", 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001903231 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270350+0000", 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0543000000000001e-05 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278988+0000", 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086371350000000006 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279079+0000", 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1012000000000002e-05 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279189+0000", 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110598 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281564+0000", 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023743929999999998 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281584+0000", 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0655000000000001e-05 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287627+0000", 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060427750000000002 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287716+0000", 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9470999999999998e-05 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287780+0000", 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3329000000000003e-05 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:45 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.449176086000001, 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019566733999999999, 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6100000000000001e-07 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268462+0000", 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6430000000000001e-06 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268469+0000", 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268475+0000", 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8039999999999998e-06 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270391+0000", 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001916713 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270397+0000", 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8490000000000002e-06 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280010+0000", 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096124780000000007 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280095+0000", 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5506000000000004e-05 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280235+0000", 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000139405 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281596+0000", 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001361615 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281676+0000", 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0259999999999994e-05 2024-09-13T11:33:43.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287969+0000", 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062924000000000001 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287999+0000", 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0270999999999999e-05 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288025+0000", 2024-09-13T11:33:43.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5488999999999999e-05 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:26 3.1 3:a6c449f0:::2fa16cfaf91e48313b717fb6e16c839aad29b3524900af3b2b4692c59d034ccb:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.448950717000002, 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019443548000000001, 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.834 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:43.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7500000000000001e-07 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268686+0000", 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1270000000000001e-06 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268689+0000", 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8709999999999999e-06 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268694+0000", 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4530000000000004e-06 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270498+0000", 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00180423 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270502+0000", 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.625e-06 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280882+0000", 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010380263000000001 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280973+0000", 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1199000000000006e-05 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281088+0000", 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011427800000000001 2024-09-13T11:33:43.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282421+0000", 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013333590000000001 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282432+0000", 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.113e-05 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288074+0000", 2024-09-13T11:33:43.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056417469999999999 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288104+0000", 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0402000000000001e-05 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288127+0000", 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2589e-05 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:28 3.1 3:96eef999:::4534b522f789f408dc72053dbc198d7d54ba3a77b9806bd26ef033725ea1a210:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.448934977, 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019504840999999998, 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:43.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6300000000000001e-07 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268702+0000", 2024-09-13T11:33:43.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.441e-06 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268709+0000", 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.888e-06 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268714+0000", 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.519e-06 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270524+0000", 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018093740000000001 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270527+0000", 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.484e-06 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281702+0000", 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011174584 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281828+0000", 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012597699999999999 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281975+0000", 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014744000000000001 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282783+0000", 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080754199999999996 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282862+0000", 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9326999999999995e-05 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288155+0000", 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005293056 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288183+0000", 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8152000000000001e-05 2024-09-13T11:33:43.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288204+0000", 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0693999999999999e-05 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:43.847 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:33:43.851 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:33:43.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1809: finishing write tid 1 to smithi20438004-8 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1809: finishing write tid 2 to smithi20438004-8 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1808: done (5 left) 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1814: read oid 5 snap -1 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1814: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1815: read oid 43 snap -1 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1815: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1816: read oid 17 snap -1 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1816: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1817: read oid 30 snap -1 2024-09-13T11:33:43.853 INFO:tasks.rados.rados.0.smithi204.stdout:1817: expect (ObjNum 370 snap 0 seq_num 370) 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1818: tier_evict oid 20 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1818: tier_evict oid 20 snap -1 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1809: finishing write tid 3 to smithi20438004-8 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1809: finishing write tid 4 to smithi20438004-8 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1809: oid 8 updating version 0 to 57 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1809: oid 8 version 57 is already newer than 55 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1809: oid 8 version 57 is already newer than 56 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 57 (ObjNum 383 snap 0 seq_num 383) dirty exists 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1809: left oid 8 (ObjNum 383 snap 0 seq_num 383) 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:1810: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3520095 2024-09-13T11:33:43.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1034895 2024-09-13T11:33:43.855 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-13T11:33:42.947668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":228912874,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":56,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":4532,"num_read_kb":2965905,"num_write":2921,"num_write_kb":671427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":112279409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":98,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":868040,"kb_used_data":522952,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1131594040,"statfs":{"total":1159641169920,"available":1158752296960,"internally_reserved":0,"allocated":535502848,"data_stored":531337194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19137,"internal_metadata":353154367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2103640,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":-4,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":613,"num_read_kb":512653,"num_write":259,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001388"},"pg_stats":[{"pgid":"4.19","version":"53'20","reported_seq":201,"reported_epoch":85,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:41.465909+0000","last_change":"2024-09-13T11:33:12.028705+0000","last_active":"2024-09-13T11:33:41.465909+0000","last_peered":"2024-09-13T11:33:41.465909+0000","last_clean":"2024-09-13T11:33:41.465909+0000","last_became_active":"2024-09-13T11:32:59.281878+0000","last_became_peered":"2024-09-13T11:32:59.281878+0000","last_unstale":"2024-09-13T11:33:41.465909+0000","last_undegraded":"2024-09-13T11:33:41.465909+0000","last_fullsized":"2024-09-13T11:33:41.465909+0000","mapping_epoch":61,"log_start":"51'17","ondisk_log_start":"51'17","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'20","last_scrub_stamp":"2024-09-13T11:33:12.028588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.028588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:17.408780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6,0,7],"avail_no_missing":["6","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"85'85","reported_seq":329,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.939603+0000","last_change":"2024-09-13T11:33:16.278769+0000","last_active":"2024-09-13T11:33:41.939603+0000","last_peered":"2024-09-13T11:33:41.939603+0000","last_clean":"2024-09-13T11:33:41.939603+0000","last_became_active":"2024-09-13T11:33:16.278471+0000","last_became_peered":"2024-09-13T11:33:16.278471+0000","last_unstale":"2024-09-13T11:33:41.939603+0000","last_undegraded":"2024-09-13T11:33:41.939603+0000","last_fullsized":"2024-09-13T11:33:41.939603+0000","mapping_epoch":72,"log_start":"80'82","ondisk_log_start":"80'82","created":56,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'57","last_scrub_stamp":"2024-09-13T11:33:04.555283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.555283+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:32.651867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176655,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":61448,"num_write":32,"num_write_kb":7957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,6],"acting":[11,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.813869+0000","last_change":"2024-09-13T11:33:03.196131+0000","last_active":"2024-09-13T11:33:41.813869+0000","last_peered":"2024-09-13T11:33:41.813869+0000","last_clean":"2024-09-13T11:33:41.813869+0000","last_became_active":"2024-09-13T11:32:56.278133+0000","last_became_peered":"2024-09-13T11:32:56.278133+0000","last_unstale":"2024-09-13T11:33:41.813869+0000","last_undegraded":"2024-09-13T11:33:41.813869+0000","last_fullsized":"2024-09-13T11:33:41.813869+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.196058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:32.709794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6],"acting":[14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.16","version":"83'94","reported_seq":352,"reported_epoch":85,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:41.825625+0000","last_change":"2024-09-13T11:32:56.273159+0000","last_active":"2024-09-13T11:33:41.825625+0000","last_peered":"2024-09-13T11:33:41.825625+0000","last_clean":"2024-09-13T11:32:55.237210+0000","last_became_active":"2024-09-13T11:32:56.273159+0000","last_became_peered":"2024-09-13T11:32:56.273159+0000","last_unstale":"2024-09-13T11:33:41.825625+0000","last_undegraded":"2024-09-13T11:32:56.268297+0000","last_fullsized":"2024-09-13T11:32:56.268083+0000","mapping_epoch":58,"log_start":"80'91","ondisk_log_start":"80'91","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":90,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:43.197862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1537890,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":182464,"num_write":71,"num_write_kb":21610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":1}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"85'51","reported_seq":265,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.844075+0000","last_change":"2024-09-13T11:33:05.991060+0000","last_active":"2024-09-13T11:33:41.844075+0000","last_peered":"2024-09-13T11:33:41.844075+0000","last_clean":"2024-09-13T11:33:41.844075+0000","last_became_active":"2024-09-13T11:32:59.285266+0000","last_became_peered":"2024-09-13T11:32:59.285266+0000","last_unstale":"2024-09-13T11:33:41.844075+0000","last_undegraded":"2024-09-13T11:33:41.844075+0000","last_fullsized":"2024-09-13T11:33:41.844075+0000","mapping_epoch":61,"log_start":"82'48","ondisk_log_start":"82'48","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'41","last_scrub_stamp":"2024-09-13T11:33:05.990981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.990981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:30.628856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016906,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":11788,"num_write":10,"num_write_kb":2641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,6],"acting":[0,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"84'117","reported_seq":423,"reported_epoch":85,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:41.465139+0000","last_change":"2024-09-13T11:33:04.630617+0000","last_active":"2024-09-13T11:33:41.465139+0000","last_peered":"2024-09-13T11:33:41.465139+0000","last_clean":"2024-09-13T11:33:41.465139+0000","last_became_active":"2024-09-13T11:32:58.287813+0000","last_became_peered":"2024-09-13T11:32:58.287813+0000","last_unstale":"2024-09-13T11:33:41.465139+0000","last_undegraded":"2024-09-13T11:33:41.465139+0000","last_fullsized":"2024-09-13T11:33:41.465139+0000","mapping_epoch":60,"log_start":"84'114","ondisk_log_start":"84'114","created":56,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'78","last_scrub_stamp":"2024-09-13T11:33:04.630512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.630512+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":114,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:28.662873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185075,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":115829,"num_write":42,"num_write_kb":11292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,3],"avail_no_missing":["4","3","7"],"object_location_counts":[{"shards":"3,4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"53'41","reported_seq":224,"reported_epoch":83,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:38.439992+0000","last_change":"2024-09-13T11:33:05.531596+0000","last_active":"2024-09-13T11:33:38.439992+0000","last_peered":"2024-09-13T11:33:38.439992+0000","last_clean":"2024-09-13T11:33:38.439992+0000","last_became_active":"2024-09-13T11:32:59.285096+0000","last_became_peered":"2024-09-13T11:32:59.285096+0000","last_unstale":"2024-09-13T11:33:38.439992+0000","last_undegraded":"2024-09-13T11:33:38.439992+0000","last_fullsized":"2024-09-13T11:33:38.439992+0000","mapping_epoch":61,"log_start":"50'38","ondisk_log_start":"50'38","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'41","last_scrub_stamp":"2024-09-13T11:33:05.531507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.531507+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:13.447752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,14],"acting":[7,14,6],"avail_no_missing":["7","6","14"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"85'135","reported_seq":451,"reported_epoch":85,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:42.161943+0000","last_change":"2024-09-13T11:33:05.196910+0000","last_active":"2024-09-13T11:33:42.161943+0000","last_peered":"2024-09-13T11:33:42.161943+0000","last_clean":"2024-09-13T11:33:42.161943+0000","last_became_active":"2024-09-13T11:32:59.288348+0000","last_became_peered":"2024-09-13T11:32:59.288348+0000","last_unstale":"2024-09-13T11:33:42.161943+0000","last_undegraded":"2024-09-13T11:33:42.161943+0000","last_fullsized":"2024-09-13T11:33:42.161943+0000","mapping_epoch":61,"log_start":"84'132","ondisk_log_start":"84'132","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'93","last_scrub_stamp":"2024-09-13T11:33:05.196778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.196778+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":132,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:09.913125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815420,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":130,"num_read_kb":128620,"num_write":50,"num_write_kb":14070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8],"acting":[14,8,3],"avail_no_missing":["14","3","8"],"object_location_counts":[{"shards":"3,8,14","objects":3}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"83'143","reported_seq":496,"reported_epoch":83,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:38.941631+0000","last_change":"2024-09-13T11:33:03.598922+0000","last_active":"2024-09-13T11:33:38.941631+0000","last_peered":"2024-09-13T11:33:38.941631+0000","last_clean":"2024-09-13T11:33:38.941631+0000","last_became_active":"2024-09-13T11:32:57.287130+0000","last_became_peered":"2024-09-13T11:32:57.287130+0000","last_unstale":"2024-09-13T11:33:38.941631+0000","last_undegraded":"2024-09-13T11:33:38.941631+0000","last_fullsized":"2024-09-13T11:33:38.941631+0000","mapping_epoch":59,"log_start":"81'140","ondisk_log_start":"81'140","created":56,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'103","last_scrub_stamp":"2024-09-13T11:33:03.598827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.598827+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":139,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:29.147515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10538088,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":175,"num_read_kb":201859,"num_write":37,"num_write_kb":14525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,15],"avail_no_missing":["7","3","15"],"object_location_counts":[{"shards":"3,7,15","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"76'68","reported_seq":320,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:42.176625+0000","last_change":"2024-09-13T11:33:12.208901+0000","last_active":"2024-09-13T11:33:42.176625+0000","last_peered":"2024-09-13T11:33:42.176625+0000","last_clean":"2024-09-13T11:33:42.176625+0000","last_became_active":"2024-09-13T11:32:59.280441+0000","last_became_peered":"2024-09-13T11:32:59.280441+0000","last_unstale":"2024-09-13T11:33:42.176625+0000","last_undegraded":"2024-09-13T11:33:42.176625+0000","last_fullsized":"2024-09-13T11:33:42.176625+0000","mapping_epoch":61,"log_start":"76'65","ondisk_log_start":"76'65","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'60","last_scrub_stamp":"2024-09-13T11:33:12.208834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.208834+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:21.081399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":49362,"num_write":10,"num_write_kb":2746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,3],"acting":[10,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":82,"state":"peering","last_fresh":"2024-09-13T11:33:35.484163+0000","last_change":"2024-09-13T11:33:38.437767+0000","last_active":"2024-09-13T11:33:38.437767+0000","last_peered":"2024-09-13T11:33:35.484163+0000","last_clean":"2024-09-13T11:33:35.484163+0000","last_became_active":"2024-09-13T11:32:45.277643+0000","last_became_peered":"2024-09-13T11:32:45.277643+0000","last_unstale":"2024-09-13T11:33:35.484163+0000","last_undegraded":"2024-09-13T11:33:35.484163+0000","last_fullsized":"2024-09-13T11:33:35.484163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0,7],"acting":[15,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6,7],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.1","version":"79'122","reported_seq":435,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-13T11:33:38.437408+0000","last_change":"2024-09-13T11:33:01.310446+0000","last_active":"2024-09-13T11:33:38.437408+0000","last_peered":"2024-09-13T11:33:38.437408+0000","last_clean":"2024-09-13T11:33:38.437408+0000","last_became_active":"2024-09-13T11:32:54.604852+0000","last_became_peered":"2024-09-13T11:32:54.604852+0000","last_unstale":"2024-09-13T11:33:38.437408+0000","last_undegraded":"2024-09-13T11:33:38.437408+0000","last_fullsized":"2024-09-13T11:33:38.437408+0000","mapping_epoch":52,"log_start":"75'119","ondisk_log_start":"75'119","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'105","last_scrub_stamp":"2024-09-13T11:33:01.310380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.310380+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":117,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:08.141302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5277464,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":72555,"num_write":16,"num_write_kb":6395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3,7],"acting":[15,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":143,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.813868+0000","last_change":"2024-09-13T11:33:39.971540+0000","last_active":"2024-09-13T11:33:41.813868+0000","last_peered":"2024-09-13T11:33:41.813868+0000","last_clean":"2024-09-13T11:33:41.813868+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:33:41.813868+0000","last_undegraded":"2024-09-13T11:33:41.813868+0000","last_fullsized":"2024-09-13T11:33:41.813868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:39.971482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:39.971482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:55.187895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"85'88","reported_seq":346,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.952368+0000","last_change":"2024-09-13T11:33:02.247329+0000","last_active":"2024-09-13T11:33:41.952368+0000","last_peered":"2024-09-13T11:33:41.952368+0000","last_clean":"2024-09-13T11:33:41.952368+0000","last_became_active":"2024-09-13T11:32:54.604560+0000","last_became_peered":"2024-09-13T11:32:54.604560+0000","last_unstale":"2024-09-13T11:33:41.952368+0000","last_undegraded":"2024-09-13T11:33:41.952368+0000","last_fullsized":"2024-09-13T11:33:41.952368+0000","mapping_epoch":51,"log_start":"82'85","ondisk_log_start":"82'85","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'57","last_scrub_stamp":"2024-09-13T11:33:02.247255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.247255+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":85,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:19.872698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6117847,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":108678,"num_write":36,"num_write_kb":10710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":85,"state":"active+undersized","last_fresh":"2024-09-13T11:33:41.810069+0000","last_change":"2024-09-13T11:32:32.498855+0000","last_active":"2024-09-13T11:33:41.810069+0000","last_peered":"2024-09-13T11:33:41.810069+0000","last_clean":"2024-09-13T11:32:30.479480+0000","last_became_active":"2024-09-13T11:32:32.498855+0000","last_became_peered":"2024-09-13T11:32:32.498855+0000","last_unstale":"2024-09-13T11:33:41.810069+0000","last_undegraded":"2024-09-13T11:33:41.810069+0000","last_fullsized":"2024-09-13T11:32:32.497672+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":["8","6"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"85'66","reported_seq":254,"reported_epoch":85,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:41.927741+0000","last_change":"2024-09-13T11:33:34.724394+0000","last_active":"2024-09-13T11:33:41.927741+0000","last_peered":"2024-09-13T11:33:41.927741+0000","last_clean":"2024-09-13T11:32:49.952102+0000","last_became_active":"2024-09-13T11:32:57.283497+0000","last_became_peered":"2024-09-13T11:32:57.283497+0000","last_unstale":"2024-09-13T11:33:41.927741+0000","last_undegraded":"2024-09-13T11:33:34.716632+0000","last_fullsized":"2024-09-13T11:32:57.277969+0000","mapping_epoch":59,"log_start":"84'63","ondisk_log_start":"84'63","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":63,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:42.921650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2168202,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":77,"num_read_kb":49555,"num_write":49,"num_write_kb":15483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2383182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":["11","6"],"object_location_counts":[{"shards":"6,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":85,"state":"active+undersized","last_fresh":"2024-09-13T11:33:41.465874+0000","last_change":"2024-09-13T11:32:57.283324+0000","last_active":"2024-09-13T11:33:41.465874+0000","last_peered":"2024-09-13T11:33:41.465874+0000","last_clean":"2024-09-13T11:32:55.276320+0000","last_became_active":"2024-09-13T11:32:57.283324+0000","last_became_peered":"2024-09-13T11:32:57.283324+0000","last_unstale":"2024-09-13T11:33:41.465874+0000","last_undegraded":"2024-09-13T11:33:41.465874+0000","last_fullsized":"2024-09-13T11:32:57.273798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:42.928484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"82'143","reported_seq":422,"reported_epoch":85,"state":"active+undersized+remapped","last_fresh":"2024-09-13T11:33:41.465784+0000","last_change":"2024-09-13T11:32:57.284225+0000","last_active":"2024-09-13T11:33:41.465784+0000","last_peered":"2024-09-13T11:33:41.465784+0000","last_clean":"2024-09-13T11:32:55.273515+0000","last_became_active":"2024-09-13T11:32:57.284225+0000","last_became_peered":"2024-09-13T11:32:57.284225+0000","last_unstale":"2024-09-13T11:33:41.465784+0000","last_undegraded":"2024-09-13T11:33:41.465784+0000","last_fullsized":"2024-09-13T11:32:57.278637+0000","mapping_epoch":59,"log_start":"81'140","ondisk_log_start":"81'140","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":137,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:45.918523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6185058,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":192,"num_read_kb":156717,"num_write":111,"num_write_kb":33400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4501279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3,9],"avail_no_missing":["3","9"],"object_location_counts":[{"shards":"3,9","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"83'303","reported_seq":506,"reported_epoch":83,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:38.492445+0000","last_change":"2024-09-13T11:33:29.491664+0000","last_active":"2024-09-13T11:33:38.492445+0000","last_peered":"2024-09-13T11:33:38.492445+0000","last_clean":"2024-09-13T11:33:38.492445+0000","last_became_active":"2024-09-13T11:32:48.281772+0000","last_became_peered":"2024-09-13T11:32:48.281772+0000","last_unstale":"2024-09-13T11:33:38.492445+0000","last_undegraded":"2024-09-13T11:33:38.492445+0000","last_fullsized":"2024-09-13T11:33:38.492445+0000","mapping_epoch":52,"log_start":"82'300","ondisk_log_start":"82'300","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'252","last_scrub_stamp":"2024-09-13T11:33:29.491435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:33:29.491435+0000","objects_scrubbed":42,"log_size":3,"log_dups_size":309,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:49.437842+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28348791,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":43,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":273,"num_read_kb":9017,"num_write":398,"num_write_kb":78431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":14112815,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15],"acting":[7,15,13],"avail_no_missing":["7","13","15"],"object_location_counts":[{"shards":"7,13,15","objects":43}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.465780+0000","last_change":"2024-09-13T11:33:26.770230+0000","last_active":"2024-09-13T11:33:41.465780+0000","last_peered":"2024-09-13T11:33:41.465780+0000","last_clean":"2024-09-13T11:33:41.465780+0000","last_became_active":"2024-09-13T11:32:43.554077+0000","last_became_peered":"2024-09-13T11:32:43.554077+0000","last_unstale":"2024-09-13T11:33:41.465780+0000","last_undegraded":"2024-09-13T11:33:41.465780+0000","last_fullsized":"2024-09-13T11:33:41.465780+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:33:26.770137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:26.770137+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:51.278083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,11],"acting":[3,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"75'81","reported_seq":338,"reported_epoch":85,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:41.465697+0000","last_change":"2024-09-13T11:33:01.558907+0000","last_active":"2024-09-13T11:33:41.465697+0000","last_peered":"2024-09-13T11:33:41.465697+0000","last_clean":"2024-09-13T11:33:41.465697+0000","last_became_active":"2024-09-13T11:32:54.600364+0000","last_became_peered":"2024-09-13T11:32:54.600364+0000","last_unstale":"2024-09-13T11:33:41.465697+0000","last_undegraded":"2024-09-13T11:33:41.465697+0000","last_fullsized":"2024-09-13T11:33:41.465697+0000","mapping_epoch":49,"log_start":"74'78","ondisk_log_start":"74'78","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'73","last_scrub_stamp":"2024-09-13T11:33:01.558809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.558809+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:23.044953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857223,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":57,"num_read_kb":57615,"num_write":9,"num_write_kb":2663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8,5],"avail_no_missing":["3","5","8"],"object_location_counts":[{"shards":"3,5,8","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"83'292","reported_seq":495,"reported_epoch":83,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:38.946645+0000","last_change":"2024-09-13T11:32:57.280350+0000","last_active":"2024-09-13T11:33:38.946645+0000","last_peered":"2024-09-13T11:33:38.946645+0000","last_clean":"2024-09-13T11:32:54.594742+0000","last_became_active":"2024-09-13T11:32:57.280350+0000","last_became_peered":"2024-09-13T11:32:57.280350+0000","last_unstale":"2024-09-13T11:33:38.946645+0000","last_undegraded":"2024-09-13T11:32:57.275975+0000","last_fullsized":"2024-09-13T11:32:57.275775+0000","mapping_epoch":59,"log_start":"83'289","ondisk_log_start":"83'289","created":30,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":295,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28840504,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":267,"num_read_kb":12162,"num_write":380,"num_write_kb":79630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":24388633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6],"acting":[15,6],"avail_no_missing":["15","6"],"object_location_counts":[{"shards":"6,15","objects":38}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.813773+0000","last_change":"2024-09-13T11:33:28.020586+0000","last_active":"2024-09-13T11:33:41.813773+0000","last_peered":"2024-09-13T11:33:41.813773+0000","last_clean":"2024-09-13T11:33:41.813773+0000","last_became_active":"2024-09-13T11:32:45.284134+0000","last_became_peered":"2024-09-13T11:32:45.284134+0000","last_unstale":"2024-09-13T11:33:41.813773+0000","last_undegraded":"2024-09-13T11:33:41.813773+0000","last_fullsized":"2024-09-13T11:33:41.813773+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:28.020512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:28.020512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:48.300493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,8],"acting":[14,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":151,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.825850+0000","last_change":"2024-09-13T11:33:03.236076+0000","last_active":"2024-09-13T11:33:41.825850+0000","last_peered":"2024-09-13T11:33:41.825850+0000","last_clean":"2024-09-13T11:33:41.825850+0000","last_became_active":"2024-09-13T11:32:54.601517+0000","last_became_peered":"2024-09-13T11:32:54.601517+0000","last_unstale":"2024-09-13T11:33:41.825850+0000","last_undegraded":"2024-09-13T11:33:41.825850+0000","last_fullsized":"2024-09-13T11:33:41.825850+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.236001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.236001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:06.939201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,3],"acting":[10,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"85'337","reported_seq":570,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:42.161456+0000","last_change":"2024-09-13T11:33:40.953610+0000","last_active":"2024-09-13T11:33:42.161456+0000","last_peered":"2024-09-13T11:33:42.161456+0000","last_clean":"2024-09-13T11:33:42.161456+0000","last_became_active":"2024-09-13T11:32:47.277392+0000","last_became_peered":"2024-09-13T11:32:47.277392+0000","last_unstale":"2024-09-13T11:33:42.161456+0000","last_undegraded":"2024-09-13T11:33:42.161456+0000","last_fullsized":"2024-09-13T11:33:42.161456+0000","mapping_epoch":51,"log_start":"85'334","ondisk_log_start":"85'334","created":30,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'333","last_scrub_stamp":"2024-09-13T11:33:40.953425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:33:40.953425+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":349,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:54.797020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28753749,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":293,"num_read_kb":6229,"num_write":421,"num_write_kb":92426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":28975184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,0],"acting":[6,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.465628+0000","last_change":"2024-09-13T11:33:35.914912+0000","last_active":"2024-09-13T11:33:41.465628+0000","last_peered":"2024-09-13T11:33:41.465628+0000","last_clean":"2024-09-13T11:33:41.465628+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:33:41.465628+0000","last_undegraded":"2024-09-13T11:33:41.465628+0000","last_fullsized":"2024-09-13T11:33:41.465628+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:35.914850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:35.914850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:50.604726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"85'92","reported_seq":367,"reported_epoch":85,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:42.142436+0000","last_change":"2024-09-13T11:32:57.280636+0000","last_active":"2024-09-13T11:33:42.142436+0000","last_peered":"2024-09-13T11:33:42.142436+0000","last_clean":"2024-09-13T11:32:54.904504+0000","last_became_active":"2024-09-13T11:32:57.280636+0000","last_became_peered":"2024-09-13T11:32:57.280636+0000","last_unstale":"2024-09-13T11:33:42.142436+0000","last_undegraded":"2024-09-13T11:32:57.277736+0000","last_fullsized":"2024-09-13T11:32:57.277594+0000","mapping_epoch":59,"log_start":"84'89","ondisk_log_start":"84'89","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":87,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:33:45.107106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5218722,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":235,"num_read_kb":182872,"num_write":63,"num_write_kb":20007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"83'369","reported_seq":565,"reported_epoch":83,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:38.968734+0000","last_change":"2024-09-13T11:33:16.277928+0000","last_active":"2024-09-13T11:33:38.968734+0000","last_peered":"2024-09-13T11:33:38.968734+0000","last_clean":"2024-09-13T11:32:55.257840+0000","last_became_active":"2024-09-13T11:33:16.277928+0000","last_became_peered":"2024-09-13T11:33:16.277928+0000","last_unstale":"2024-09-13T11:33:38.968734+0000","last_undegraded":"2024-09-13T11:33:16.272371+0000","last_fullsized":"2024-09-13T11:33:16.272253+0000","mapping_epoch":72,"log_start":"83'366","ondisk_log_start":"83'366","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":381,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18325013,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":334,"num_read_kb":5448,"num_write":472,"num_write_kb":91838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":16251716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3],"acting":[15,3],"avail_no_missing":["15","3"],"object_location_counts":[{"shards":"3,15","objects":34}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":114,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-13T11:33:38.437501+0000","last_change":"2024-09-13T11:33:27.080910+0000","last_active":"2024-09-13T11:33:38.437501+0000","last_peered":"2024-09-13T11:33:38.437501+0000","last_clean":"2024-09-13T11:33:38.437501+0000","last_became_active":"2024-09-13T11:32:45.284150+0000","last_became_peered":"2024-09-13T11:32:45.284150+0000","last_unstale":"2024-09-13T11:33:38.437501+0000","last_undegraded":"2024-09-13T11:33:38.437501+0000","last_fullsized":"2024-09-13T11:33:38.437501+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:33:27.080847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:27.080847+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:31.383253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,0],"acting":[15,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":253,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.843969+0000","last_change":"2024-09-13T11:33:16.944043+0000","last_active":"2024-09-13T11:33:41.843969+0000","last_peered":"2024-09-13T11:33:41.843969+0000","last_clean":"2024-09-13T11:33:41.843969+0000","last_became_active":"2024-09-13T11:32:48.286550+0000","last_became_peered":"2024-09-13T11:32:48.286550+0000","last_unstale":"2024-09-13T11:33:41.843969+0000","last_undegraded":"2024-09-13T11:33:41.843969+0000","last_fullsized":"2024-09-13T11:33:41.843969+0000","mapping_epoch":52,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:33:16.943930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:33:16.943930+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:23.725756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"82'47","reported_seq":255,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.828836+0000","last_change":"2024-09-13T11:33:02.151042+0000","last_active":"2024-09-13T11:33:41.828836+0000","last_peered":"2024-09-13T11:33:41.828836+0000","last_clean":"2024-09-13T11:33:41.828836+0000","last_became_active":"2024-09-13T11:32:54.609316+0000","last_became_peered":"2024-09-13T11:32:54.609316+0000","last_unstale":"2024-09-13T11:33:41.828836+0000","last_undegraded":"2024-09-13T11:33:41.828836+0000","last_fullsized":"2024-09-13T11:33:41.828836+0000","mapping_epoch":50,"log_start":"76'44","ondisk_log_start":"76'44","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'41","last_scrub_stamp":"2024-09-13T11:33:02.150975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.150975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":43,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:24.325318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3311012,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2028,"num_write":5,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"80'54","reported_seq":273,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.465054+0000","last_change":"2024-09-13T11:33:01.646865+0000","last_active":"2024-09-13T11:33:41.465054+0000","last_peered":"2024-09-13T11:33:41.465054+0000","last_clean":"2024-09-13T11:33:41.465054+0000","last_became_active":"2024-09-13T11:32:54.601608+0000","last_became_peered":"2024-09-13T11:32:54.601608+0000","last_unstale":"2024-09-13T11:33:41.465054+0000","last_undegraded":"2024-09-13T11:33:41.465054+0000","last_fullsized":"2024-09-13T11:33:41.465054+0000","mapping_epoch":31,"log_start":"61'51","ondisk_log_start":"61'51","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'53","last_scrub_stamp":"2024-09-13T11:33:01.646779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.646779+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:10.069513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3546277,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":71001,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"81'54","reported_seq":281,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.939525+0000","last_change":"2024-09-13T11:33:02.042709+0000","last_active":"2024-09-13T11:33:41.939525+0000","last_peered":"2024-09-13T11:33:41.939525+0000","last_clean":"2024-09-13T11:33:41.939525+0000","last_became_active":"2024-09-13T11:32:54.604804+0000","last_became_peered":"2024-09-13T11:32:54.604804+0000","last_unstale":"2024-09-13T11:33:41.939525+0000","last_undegraded":"2024-09-13T11:33:41.939525+0000","last_fullsized":"2024-09-13T11:33:41.939525+0000","mapping_epoch":31,"log_start":"78'50","ondisk_log_start":"78'50","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2024-09-13T11:33:02.042611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.042611+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":50,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:02.206008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5966551,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":117,"num_read_kb":97329,"num_write":23,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14,7],"acting":[0,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"84'74","reported_seq":292,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:41.829935+0000","last_change":"2024-09-13T11:32:47.284684+0000","last_active":"2024-09-13T11:33:41.829935+0000","last_peered":"2024-09-13T11:33:41.829935+0000","last_clean":"2024-09-13T11:33:41.829935+0000","last_became_active":"2024-09-13T11:32:47.279408+0000","last_became_peered":"2024-09-13T11:32:47.279408+0000","last_unstale":"2024-09-13T11:33:41.829935+0000","last_undegraded":"2024-09-13T11:33:41.829935+0000","last_fullsized":"2024-09-13T11:33:41.829935+0000","mapping_epoch":51,"log_start":"81'71","ondisk_log_start":"81'71","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5227076,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":171,"num_read_kb":150254,"num_write":81,"num_write_kb":21360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2979858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"83'76","reported_seq":305,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:42.177752+0000","last_change":"2024-09-13T11:32:48.288341+0000","last_active":"2024-09-13T11:33:42.177752+0000","last_peered":"2024-09-13T11:33:42.177752+0000","last_clean":"2024-09-13T11:33:42.177752+0000","last_became_active":"2024-09-13T11:32:48.283697+0000","last_became_peered":"2024-09-13T11:32:48.283697+0000","last_unstale":"2024-09-13T11:33:42.177752+0000","last_undegraded":"2024-09-13T11:33:42.177752+0000","last_fullsized":"2024-09-13T11:33:42.177752+0000","mapping_epoch":52,"log_start":"82'73","ondisk_log_start":"82'73","created":31,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":73,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:46.169371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8840614,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":222,"num_read_kb":192477,"num_write":79,"num_write_kb":23206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4770005,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"80'38","reported_seq":226,"reported_epoch":83,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:38.932123+0000","last_change":"2024-09-13T11:33:27.445927+0000","last_active":"2024-09-13T11:33:38.932123+0000","last_peered":"2024-09-13T11:33:38.932123+0000","last_clean":"2024-09-13T11:32:55.276844+0000","last_became_active":"2024-09-13T11:32:57.282855+0000","last_became_peered":"2024-09-13T11:32:57.282855+0000","last_unstale":"2024-09-13T11:33:38.932123+0000","last_undegraded":"2024-09-13T11:33:27.445134+0000","last_fullsized":"2024-09-13T11:32:57.277023+0000","mapping_epoch":59,"log_start":"78'35","ondisk_log_start":"78'35","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":35,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":68782,"num_write":35,"num_write_kb":11475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2040825,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7],"acting":[15,7],"avail_no_missing":["15","7"],"object_location_counts":[{"shards":"7,15","objects":1}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.d","version":"85'102","reported_seq":354,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-13T11:33:42.177579+0000","last_change":"2024-09-13T11:33:16.280221+0000","last_active":"2024-09-13T11:33:42.177579+0000","last_peered":"2024-09-13T11:33:42.177579+0000","last_clean":"2024-09-13T11:33:42.177579+0000","last_became_active":"2024-09-13T11:33:16.279717+0000","last_became_peered":"2024-09-13T11:33:16.279717+0000","last_unstale":"2024-09-13T11:33:42.177579+0000","last_undegraded":"2024-09-13T11:33:42.177579+0000","last_fullsized":"2024-09-13T11:33:42.177579+0000","mapping_epoch":72,"log_start":"82'99","ondisk_log_start":"82'99","created":31,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":99,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:48.989249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8807000,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":307,"num_read_kb":247876,"num_write":104,"num_write_kb":29010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"83'100","reported_seq":402,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-13T11:33:38.946645+0000","last_change":"2024-09-13T11:32:47.281013+0000","last_active":"2024-09-13T11:33:38.946645+0000","last_peered":"2024-09-13T11:33:38.946645+0000","last_clean":"2024-09-13T11:33:38.946645+0000","last_became_active":"2024-09-13T11:32:47.277862+0000","last_became_peered":"2024-09-13T11:32:47.277862+0000","last_unstale":"2024-09-13T11:33:38.946645+0000","last_undegraded":"2024-09-13T11:33:38.946645+0000","last_fullsized":"2024-09-13T11:33:38.946645+0000","mapping_epoch":51,"log_start":"82'97","ondisk_log_start":"82'97","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":97,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176754,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":379,"num_read_kb":305013,"num_write":104,"num_write_kb":32334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5522961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,0],"acting":[15,14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.f","version":"85'131","reported_seq":456,"reported_epoch":85,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:42.150939+0000","last_change":"2024-09-13T11:32:57.287440+0000","last_active":"2024-09-13T11:33:42.150939+0000","last_peered":"2024-09-13T11:33:42.150939+0000","last_clean":"2024-09-13T11:32:55.848229+0000","last_became_active":"2024-09-13T11:32:57.287440+0000","last_became_peered":"2024-09-13T11:32:57.287440+0000","last_unstale":"2024-09-13T11:33:42.150939+0000","last_undegraded":"2024-09-13T11:32:57.280313+0000","last_fullsized":"2024-09-13T11:32:57.280204+0000","mapping_epoch":59,"log_start":"85'128","ondisk_log_start":"85'128","created":31,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":128,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:46.073150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9126432,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":414,"num_read_kb":418854,"num_write":136,"num_write_kb":37382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5762564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14],"acting":[10,14],"avail_no_missing":["10","14"],"object_location_counts":[{"shards":"10,14","objects":4}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":124054430,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":3279,"num_read_kb":2932976,"num_write":1103,"num_write_kb":326428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":27960674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":28,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":309116928,"data_stored":308551634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":64,"acting":72,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":104268057,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":72,"num_objects_misplaced":43,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":1167,"num_read_kb":32856,"num_write":1671,"num_write_kb":342325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":83728348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":388685824,"data_stored":387718526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":9,"acting":10,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":22,"acting":22,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126164,"kb_used_data":94980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94245676,"statfs":{"total":96636764160,"available":96507572224,"internally_reserved":0,"allocated":97259520,"data_stored":96766055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":98784247830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247830,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87584,"kb_used_data":56400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94284256,"statfs":{"total":96636764160,"available":96547078144,"internally_reserved":0,"allocated":57753600,"data_stored":57390658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82748,"kb_used_data":51564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94289092,"statfs":{"total":96636764160,"available":96552030208,"internally_reserved":0,"allocated":52801536,"data_stored":52414977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280534,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49048,"kb_used_data":22024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322792,"statfs":{"total":96636764160,"available":96586539008,"internally_reserved":0,"allocated":22552576,"data_stored":22264951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":170144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280535,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94944,"kb_used_data":63760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94276896,"statfs":{"total":96636764160,"available":96539541504,"internally_reserved":0,"allocated":65290240,"data_stored":64919402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,4,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47624,"kb_used_data":20600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324216,"statfs":{"total":96636764160,"available":96587997184,"internally_reserved":0,"allocated":21094400,"data_stored":20804864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":94489280534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280535,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100184,"kb_used_data":69000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94271656,"statfs":{"total":96636764160,"available":96534175744,"internally_reserved":0,"allocated":70656000,"data_stored":70217686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280535,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39696,"kb_used_data":12672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332144,"statfs":{"total":96636764160,"available":96596115456,"internally_reserved":0,"allocated":12976128,"data_stored":12713390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280535,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48204,"kb_used_data":21180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323636,"statfs":{"total":96636764160,"available":96587403264,"internally_reserved":0,"allocated":21688320,"data_stored":21385458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313238,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69136,"kb_used_data":42112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94302704,"statfs":{"total":96636764160,"available":96565968896,"internally_reserved":0,"allocated":43122688,"data_stored":42734553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,3,4,6,7,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95324,"kb_used_data":68300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276516,"statfs":{"total":96636764160,"available":96539152384,"internally_reserved":0,"allocated":69939200,"data_stored":69555056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28835840,"data_stored":28753749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24444928,"data_stored":24413026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23801856,"data_stored":23781575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20439040,"data_stored":20374549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":58646528,"data_stored":58484621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28434432,"data_stored":28348791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28405760,"data_stored":28348291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28835840,"data_stored":28753749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":42668032,"data_stored":42597076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":28434432,"data_stored":28348791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":75739136,"data_stored":75514308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25042944,"data_stored":24989748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18067456,"data_stored":18030231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4225584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46718976,"data_stored":46655961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20348928,"data_stored":20305601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2412544,"data_stored":2410759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14426112,"data_stored":14391197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":29540352,"data_stored":29498271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12218368,"data_stored":12194721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5033984,"data_stored":5030794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":21364736,"data_stored":21309491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16273408,"data_stored":16215602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":26669056,"data_stored":26630598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":24473600,"data_stored":24444051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":21487616,"data_stored":21449759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":20799488,"data_stored":20769266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:33:43.857 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-13T11:33:43.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-09-13T11:33:43.858 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:43.847+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:43.914 INFO:tasks.rados.rados.0.smithi204.stdout:1811: expect (ObjNum 322 snap 0 seq_num 322) 2024-09-13T11:33:43.949 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:43.948+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:43.995 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:43.995 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:43.995 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:43.995 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:44.004 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:44.050 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:44.048+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:44.092 INFO:tasks.rados.rados.0.smithi204.stdout:1812: finishing write tid 1 to smithi20438004-10 2024-09-13T11:33:44.092 INFO:tasks.rados.rados.0.smithi204.stdout:1813: got (22) Invalid argument 2024-09-13T11:33:44.092 INFO:tasks.rados.rados.0.smithi204.stdout:1812: finishing write tid 2 to smithi20438004-10 2024-09-13T11:33:44.092 INFO:tasks.rados.rados.0.smithi204.stdout:1812: finishing write tid 3 to smithi20438004-10 2024-09-13T11:33:44.092 INFO:tasks.rados.rados.0.smithi204.stdout:1812: finishing write tid 4 to smithi20438004-10 2024-09-13T11:33:44.092 INFO:tasks.rados.rados.0.smithi204.stdout:1812: oid 10 updating version 0 to 126 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1812: oid 10 version 126 is already newer than 125 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1812: oid 10 version 126 is already newer than 124 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 126 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1812: left oid 10 (ObjNum 384 snap 0 seq_num 384) 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1809: done (9 left) 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1810: done (8 left) 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1811: done (7 left) 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1812: done (6 left) 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1813: done (5 left) 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1819: read oid 15 snap -1 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1819: expect (ObjNum 373 snap 0 seq_num 373) 2024-09-13T11:33:44.093 INFO:tasks.rados.rados.0.smithi204.stdout:1820: tier_promote oid 22 2024-09-13T11:33:44.094 INFO:tasks.rados.rados.0.smithi204.stdout:1818: got (22) Invalid argument 2024-09-13T11:33:44.094 INFO:tasks.rados.rados.0.smithi204.stdout:1816: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:44.094 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:44.107 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:44.107 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:44.107 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:44.107 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:44.107 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:44.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:44.161 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:44.160+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:44.222 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:44.222 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:254 4.0 4:04aeeab9:::smithi20438004-5:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.018410485999993, 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022108323999999999, 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.223 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 254 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:44.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201594+0000", 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201603+0000", 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.568e-06 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201619+0000", 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6540000000000001e-05 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201629+0000", 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3200000000000006e-06 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201685+0000", 2024-09-13T11:33:44.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6078000000000001e-05 2024-09-13T11:33:44.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201747+0000", 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2242999999999993e-05 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220103+0000", 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01835548 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220937+0000", 2024-09-13T11:33:44.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083429299999999995 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221938+0000", 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010010710000000001 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223567+0000", 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001628846 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223638+0000", 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1165000000000001e-05 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223701+0000", 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3120999999999994e-05 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:285 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:44.230 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.985426027000003, 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012191677, 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 285 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.231 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234578+0000", 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234583+0000", 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9110000000000001e-06 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234593+0000", 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234598+0000", 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9509999999999997e-06 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234639+0000", 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1826999999999998e-05 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234688+0000", 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9039000000000003e-05 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239203+0000", 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045143739999999998 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239979+0000", 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077664100000000001 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.242634+0000", 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026544899999999998 2024-09-13T11:33:44.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246683+0000", 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040491499999999996 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246723+0000", 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0429000000000002e-05 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246769+0000", 2024-09-13T11:33:44.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5430999999999999e-05 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1715 4.f 4:fea8c5b2:::smithi20438004-39:head [write 769787~704463,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.951617087000002, 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020210576000000001, 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.237 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1715 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268388+0000", 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269172+0000", 2024-09-13T11:33:44.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078478 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269184+0000", 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.182e-05 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269194+0000", 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8570000000000007e-06 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269269+0000", 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4795000000000005e-05 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269283+0000", 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275426+0000", 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061428050000000003 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275540+0000", 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011432199999999999 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275658+0000", 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118071 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278252+0000", 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594224 2024-09-13T11:33:44.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279142+0000", 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088964600000000001 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288321+0000", 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091785600000000005 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288363+0000", 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2957000000000002e-05 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288492+0000", 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012896899999999999 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288597+0000", 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010429000000000001 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.244 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1716 4.f 4:fea8c5b2:::smithi20438004-39:head [write 2267961~101826] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.950790230000003, 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019610079999999998, 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1716 2024-09-13T11:33:44.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269214+0000", 2024-09-13T11:33:44.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9699999999999995e-07 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269305+0000", 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1330999999999997e-05 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269314+0000", 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9209999999999998e-06 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269321+0000", 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269363+0000", 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2388999999999997e-05 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269372+0000", 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277175+0000", 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078032580000000004 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277247+0000", 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1773999999999994e-05 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277354+0000", 2024-09-13T11:33:44.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010704000000000001 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288308+0000", 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010953353000000001 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288328+0000", 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0353e-05 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288640+0000", 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031178999999999999 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288706+0000", 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6749999999999996e-05 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288765+0000", 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8876999999999999e-05 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288823+0000", 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7806999999999997e-05 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.252 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1717 4.f 4:fea8c5b2:::smithi20438004-39:head [setxattr _header (56) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.950665968000003, 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019744788999999999, 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1717 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269338+0000", 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269342+0000", 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.924e-06 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269349+0000", 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9619999999999996e-06 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269355+0000", 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1210000000000003e-06 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269399+0000", 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4360000000000002e-05 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5570000000000001e-06 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278293+0000", 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088867549999999997 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278372+0000", 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9561999999999995e-05 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278492+0000", 2024-09-13T11:33:44.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119654 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288335+0000", 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098426840000000008 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288672+0000", 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033785899999999998 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288863+0000", 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019105599999999999 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288904+0000", 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0738000000000002e-05 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288949+0000", 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4443000000000001e-05 2024-09-13T11:33:44.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289082+0000", 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013346199999999999 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.260 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1718 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.950636494000001, 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019946973, 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1718 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.261 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:44.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8999999999999996e-07 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269373+0000", 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8629999999999996e-06 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269379+0000", 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4979999999999997e-06 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269383+0000", 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7260000000000002e-06 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269443+0000", 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9616e-05 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269451+0000", 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279055+0000", 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009604586 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279104+0000", 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8131999999999999e-05 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289119+0000", 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010015616 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289177+0000", 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8060999999999999e-05 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289314+0000", 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000136525 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.266 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1732 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.950611999000003, 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017069699000000001, 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.267 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1732 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269392+0000", 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269395+0000", 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-06 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269400+0000", 2024-09-13T11:33:44.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.198e-06 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9410000000000004e-06 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269514+0000", 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108124 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269524+0000", 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279170+0000", 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096457209999999995 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279248+0000", 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8147999999999998e-05 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286461+0000", 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072129789999999996 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1733 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.950569023, 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0187283, 2024-09-13T11:33:44.272 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1733 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269435+0000", 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7800000000000001e-07 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269438+0000", 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.535e-06 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269447+0000", 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6559999999999998e-06 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269452+0000", 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.972e-06 2024-09-13T11:33:44.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269552+0000", 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9820000000000003e-05 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269556+0000", 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:33:44.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286496+0000", 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016940182000000002 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286550+0000", 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3640999999999997e-05 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288163+0000", 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016126720000000001 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1734 4.f 4:f653a832:::smithi20438004-19:head [read 1042153~2571319,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.950503122000001, 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021098286000000001, 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.278 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1734 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269501+0000", 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269504+0000", 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.58e-06 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269511+0000", 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6200000000000001e-06 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269517+0000", 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7030000000000003e-06 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269580+0000", 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:33:44.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269583+0000", 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5020000000000001e-06 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289341+0000", 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019757429 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289385+0000", 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3939000000000001e-05 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.290598+0000", 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012136130000000001 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.12.0:167 4.6 e51/31 4:6978b14d:::smithi20438004-23:head v 51'52, mlcod=45'50)", 2024-09-13T11:33:44.283 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.028388778999997, 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013003298, 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191616+0000", 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0270000000000001e-06 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196708+0000", 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050925720000000001 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196728+0000", 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9616999999999999e-05 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196738+0000", 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8330000000000006e-06 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196817+0000", 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9225000000000001e-05 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196841+0000", 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3972e-05 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204573+0000", 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077323629999999999 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204618+0000", 2024-09-13T11:33:44.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4688999999999999e-05 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1770 4.0 4:001d90c2:::smithi20438004-28:head [read 0~2218524,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.950214160999998, 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011933050000000001, 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.288 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1770 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269791+0000", 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269795+0000", 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269803+0000", 2024-09-13T11:33:44.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269810+0000", 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.246e-06 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273037+0000", 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032276700000000002 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273048+0000", 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0511e-05 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.277997+0000", 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049497539999999998 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278150+0000", 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152406 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281722+0000", 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00357224 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1771 4.0 4:001d90c2:::smithi20438004-28:head [read 633380~1585144,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.950184157000002, 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013022562, 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.293 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1771 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269820+0000", 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4000000000000002e-07 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269821+0000", 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9269999999999999e-06 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269825+0000", 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8789999999999996e-06 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269829+0000", 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4740000000000001e-06 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273144+0000", 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033155459999999999 2024-09-13T11:33:44.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273155+0000", 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.062e-05 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282280+0000", 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091252620000000003 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282403+0000", 2024-09-13T11:33:44.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012240499999999999 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282842+0000", 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043900900000000002 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1772 4.0 4:001d90c2:::smithi20438004-28:head [sparse-read 2038646~179878] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.950170712000002, 2024-09-13T11:33:44.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013177762000000001, 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1772 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9499999999999998e-07 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269834+0000", 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269837+0000", 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.005e-06 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269843+0000", 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6389999999999997e-06 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273188+0000", 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033454840000000001 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273196+0000", 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0830000000000004e-06 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282874+0000", 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096776810000000005 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282924+0000", 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0095000000000001e-05 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.283010+0000", 2024-09-13T11:33:44.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6216000000000003e-05 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2324 4.16 4:6978b14d:::smithi20438004-23:head [write 544593~799729,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 48.082922730999996, 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14427186, 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2324 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:44.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7100000000000001e-07 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137660+0000", 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057879000000000003 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137671+0000", 2024-09-13T11:33:44.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0518e-05 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137675+0000", 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137754+0000", 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8492000000000007e-05 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137765+0000", 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1027e-05 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273433+0000", 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13566771999999999 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273569+0000", 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013616999999999999 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273699+0000", 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130142 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281227+0000", 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075284699999999998 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281259+0000", 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1610999999999999e-05 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281298+0000", 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9138999999999999e-05 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281352+0000", 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2325 4.16 4:6978b14d:::smithi20438004-23:head [write 1761799~776065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 48.076257415000001, 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13997515799999999, 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2325 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143748+0000", 2024-09-13T11:33:44.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000001e-06 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144385+0000", 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000637125 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144396+0000", 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0633e-05 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144401+0000", 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2469999999999999e-06 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144457+0000", 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5566999999999998e-05 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144467+0000", 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8819999999999996e-06 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275371+0000", 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13090408000000001 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275440+0000", 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8749000000000002e-05 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275530+0000", 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0236999999999997e-05 2024-09-13T11:33:44.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282215+0000", 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066849240000000001 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283575+0000", 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001360775 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283629+0000", 2024-09-13T11:33:44.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283721+0000", 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1799999999999995e-05 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2326 4.16 4:6978b14d:::smithi20438004-23:head [write 2948352~396241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:44.317 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 48.071891993999998, 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13583619799999999, 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2326 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148113+0000", 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:33:44.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148305+0000", 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000192008 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148316+0000", 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0293e-05 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148327+0000", 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1489e-05 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148398+0000", 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0421999999999998e-05 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148407+0000", 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1090000000000004e-06 2024-09-13T11:33:44.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279356+0000", 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13094884700000001 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279444+0000", 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279534+0000", 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9573000000000005e-05 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282581+0000", 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030473990000000001 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283856+0000", 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001275307 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283901+0000", 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5238999999999998e-05 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283947+0000", 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5972e-05 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2327 4.16 4:6978b14d:::smithi20438004-23:head [setxattr _header (56) in=7b,truncate 3344593] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 48.071664493999997, 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13580746099999999, 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.325 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2327 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:44.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.34e-07 2024-09-13T11:33:44.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148341+0000", 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0999999999999998e-06 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148363+0000", 2024-09-13T11:33:44.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2045999999999999e-05 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148369+0000", 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.665e-06 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148463+0000", 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4475999999999995e-05 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148474+0000", 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0319999999999999e-05 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280931+0000", 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132457142 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280994+0000", 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2693000000000004e-05 2024-09-13T11:33:44.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281088+0000", 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3993000000000006e-05 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282632+0000", 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001544199 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283989+0000", 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013571670000000001 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284048+0000", 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9070999999999997e-05 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284146+0000", 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8255e-05 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.332 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2328 4.16 4:6978b14d:::smithi20438004-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 48.071629979000001, 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13592306100000001, 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.333 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2328 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148374+0000", 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6899999999999999e-07 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148375+0000", 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148378+0000", 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0989999999999999e-06 2024-09-13T11:33:44.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148381+0000", 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148500+0000", 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119052 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:44.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148509+0000", 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8759999999999994e-06 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281377+0000", 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132868244 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281422+0000", 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4909e-05 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284171+0000", 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027497300000000001 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284216+0000", 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4626999999999998e-05 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284296+0000", 2024-09-13T11:33:44.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0275e-05 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2687 4.10 4:0c9026a3:::smithi20438004-36:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:44.339 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 38.148177795000002, 2024-09-13T11:33:44.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029783952999999998, 2024-09-13T11:33:44.341 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.341 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.341 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2687 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071827+0000", 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.096e-06 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071831+0000", 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8210000000000001e-06 2024-09-13T11:33:44.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071843+0000", 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1715e-05 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071848+0000", 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071899+0000", 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0290000000000001e-05 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071988+0000", 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.101609+0000", 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0296212 2024-09-13T11:33:44.345 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3509 4.f 4:f5aacf61:::smithi20438004-27:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.440658618000001, 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022491955000000001, 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.346 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3509 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779347+0000", 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9209999999999999e-06 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779354+0000", 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7980000000000006e-06 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779373+0000", 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289999999999999e-05 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779399+0000", 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779457+0000", 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8888000000000003e-05 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779523+0000", 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5995999999999996e-05 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.801837+0000", 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022313116000000001 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.350 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.351 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.351 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.351 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.351 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:44.352 INFO:tasks.rados.rados.0.smithi204.stdout:1815: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:44.352 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:44.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:44.354 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:44.351+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:44.428 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 867245 2024-09-13T11:33:44.453 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:44.451+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:44.469 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pg_num to 36 2024-09-13T11:33:44.469 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:33:44.487 INFO:tasks.rados.rados.0.smithi204.stdout:1817: expect (ObjNum 370 snap 0 seq_num 370) 2024-09-13T11:33:44.500 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:44.500 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:44.500 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:44.500 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:44.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:44.535 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:44.554 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:44.552+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:44.613 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:44.613 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:44.613 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:44.613 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:44.613 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:44.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:44.628 INFO:tasks.rados.rados.0.smithi204.stdout:1814: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-13T11:33:44.654 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:44.653+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:44.725 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.328830733000004, 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078075499999999999, 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394848+0000", 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397492+0000", 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026441139999999999 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397513+0000", 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0381e-05 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397520+0000", 2024-09-13T11:33:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4789999999999999e-06 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397586+0000", 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5687999999999999e-05 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397612+0000", 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5870000000000001e-05 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402610+0000", 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049984199999999999 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402654+0000", 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4153000000000002e-05 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.692425641, 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081668139999999997, 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:33:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031253+0000", 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7900000000000007e-07 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037708+0000", 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064553029999999999 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037720+0000", 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2062000000000001e-05 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037745+0000", 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4766999999999999e-05 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037797+0000", 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2198999999999999e-05 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037815+0000", 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8178000000000001e-05 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039386+0000", 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015708110000000001 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039419+0000", 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2515000000000003e-05 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.4631.0:1715 4.f e51/50 4:fea8c5b2:::smithi20438004-39:head v 51'67, mlcod=0'0)", 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 58.447908302999998, 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063741700000000002, 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275769+0000", 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275770+0000", 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9999999999999997e-07 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276160+0000", 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038996700000000002 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276171+0000", 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0702e-05 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276177+0000", 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7750000000000004e-06 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276230+0000", 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2751999999999999e-05 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.276252+0000", 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2055000000000001e-05 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282092+0000", 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058396180000000004 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.282143+0000", 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1601000000000003e-05 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4d4c8512:::smithi20438004-29:head, version: 55'82, data_included: [405504~978944,1437696~69632], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4d4c8512:::smithi20438004-29:head@55'82, size: 2806769, copy_subset: [(0, 2806769)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1507328, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.389532312999997, 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064952710000000004, 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334145+0000", 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.334146+0000", 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9500000000000002e-07 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339863+0000", 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057170859999999997 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339880+0000", 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6705999999999999e-05 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339897+0000", 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7663999999999999e-05 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339963+0000", 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5808000000000004e-05 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.339972+0000", 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6289999999999999e-06 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:57.340640+0000", 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066878300000000005 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2515 4.12 4:4d4c8512:::smithi20438004-29:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 43.645681078000003, 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008644934, 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2515 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077997+0000", 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5999999999999991e-07 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078584+0000", 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058694699999999999 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078595+0000", 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0808999999999999e-05 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078600+0000", 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9779999999999996e-06 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078667+0000", 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6445999999999994e-05 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078732+0000", 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5061000000000006e-05 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078820+0000", 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8208999999999995e-05 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080991+0000", 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021715419999999998 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084271+0000", 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032798079999999999 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086533+0000", 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022614810000000001 2024-09-13T11:33:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086566+0000", 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2959000000000001e-05 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086641+0000", 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5734000000000004e-05 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2516 4.12 4:4d4c8512:::smithi20438004-29:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 43.645066370999999, 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082031369999999992, 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2516 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078612+0000", 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078631+0000", 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9823999999999999e-05 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078642+0000", 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.011e-05 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078647+0000", 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T11:33:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.079994+0000", 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001346835 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080073+0000", 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8721999999999996e-05 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080232+0000", 2024-09-13T11:33:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000159698 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.081037+0000", 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080509799999999997 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084295+0000", 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032578360000000001 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086714+0000", 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024189799999999998 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086749+0000", 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4331e-05 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086814+0000", 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5703000000000004e-05 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2517 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 43.645022605999998, 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083158850000000003, 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2517 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:44.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9800000000000004e-07 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078658+0000", 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1940000000000001e-06 2024-09-13T11:33:44.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078664+0000", 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1859999999999997e-06 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078668+0000", 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6820000000000002e-06 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080476+0000", 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018074160000000001 2024-09-13T11:33:44.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080525+0000", 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9036000000000003e-05 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086839+0000", 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063138650000000001 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086884+0000", 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5717999999999998e-05 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086971+0000", 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6290000000000002e-05 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2537 4.12 4:4b088bf8:::smithi20438004-24:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 43.604727220999997, 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077763800000000003, 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2537 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118952+0000", 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0159999999999998e-06 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118976+0000", 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4227999999999999e-05 2024-09-13T11:33:44.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118987+0000", 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119009+0000", 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2232999999999999e-05 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119058+0000", 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8427999999999998e-05 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119146+0000", 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.126727+0000", 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075806049999999998 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3033 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3663926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.225177366, 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083775140000000008, 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3033 2024-09-13T11:33:44.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4499999999999998e-07 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498503+0000", 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4889999999999998e-06 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498508+0000", 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7079999999999996e-06 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498511+0000", 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1879999999999998e-06 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503489+0000", 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049776849999999999 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503558+0000", 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9128000000000006e-05 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.506878+0000", 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033199710000000001 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.770 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3042 4.12 4:4b088bf8:::smithi20438004-24:head [write 753013~645493] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.613336276999998, 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099953939999999995, 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3042 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110345+0000", 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.563e-06 2024-09-13T11:33:44.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111084+0000", 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073883200000000003 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111101+0000", 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7039999999999999e-05 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111110+0000", 2024-09-13T11:33:44.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9060000000000008e-06 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111178+0000", 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8503000000000005e-05 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111183+0000", 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2959999999999998e-06 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112893+0000", 2024-09-13T11:33:44.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001709728 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112975+0000", 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1757999999999999e-05 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.113113+0000", 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013859500000000001 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119968+0000", 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068545869999999997 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120007+0000", 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584000000000002e-05 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120085+0000", 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8226999999999995e-05 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120188+0000", 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103243 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120337+0000", 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014853199999999999 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3045 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:44.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.608066729000001, 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060597120000000001, 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3045 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:44.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115611+0000", 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.03e-07 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115613+0000", 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.598e-06 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115616+0000", 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7500000000000001e-06 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.115619+0000", 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.089e-06 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119902+0000", 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042830250000000002 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119956+0000", 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3075999999999998e-05 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121340+0000", 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013844669999999999 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121395+0000", 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4721e-05 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.121670+0000", 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027568299999999998 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.782 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3369 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3663926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.714578408000001, 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077891109999999996, 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.783 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3369 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009100+0000", 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1900000000000002e-07 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009105+0000", 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6899999999999997e-06 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009114+0000", 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1079999999999999e-06 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009121+0000", 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5320000000000003e-06 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.010369+0000", 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001247915 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.010466+0000", 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7579999999999997e-05 2024-09-13T11:33:44.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.016888+0000", 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064216669999999998 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3370 4.12 4:4d4c8512:::smithi20438004-29:head [read 32972~3630954,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.714545180999998, 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010949787000000001, 2024-09-13T11:33:44.787 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3370 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009133+0000", 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7299999999999995e-07 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009136+0000", 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5090000000000002e-06 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009144+0000", 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2880000000000004e-06 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009150+0000", 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.016932+0000", 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077818109999999996 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.017048+0000", 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011644799999999999 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.020082+0000", 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030339849999999999 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3846 4.12 4:4ba9f189:::smithi20438004-14:head [write 497573~628943,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.587147+0000", 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.136530045000001, 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010066912000000001, 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3846 2024-09-13T11:33:44.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.587147+0000", 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.587147+0000", 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.587150+0000", 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7350000000000001e-06 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.588997+0000", 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001846952 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589014+0000", 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7011e-05 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589024+0000", 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8859999999999999e-06 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589103+0000", 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8969999999999998e-05 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589171+0000", 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7947999999999994e-05 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589277+0000", 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105714 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.593005+0000", 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037278720000000001 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.593997+0000", 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00099221599999999997 2024-09-13T11:33:44.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.597083+0000", 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003086063 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.597130+0000", 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7005999999999997e-05 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.597214+0000", 2024-09-13T11:33:44.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4538999999999998e-05 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3868 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~3546310,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.781476+0000", 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.942201859000001, 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075397750000000003, 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3868 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781476+0000", 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781476+0000", 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781477+0000", 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781488+0000", 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781503+0000", 2024-09-13T11:33:44.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4932e-05 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781510+0000", 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3010000000000001e-06 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781577+0000", 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7436999999999998e-05 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781690+0000", 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011268399999999999 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.789015+0000", 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073252109999999999 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3869 4.12 4:4b088bf8:::smithi20438004-24:head [read 2720685~825625,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.781527+0000", 2024-09-13T11:33:44.802 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.942150389, 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085861070000000008, 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3869 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781527+0000", 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781527+0000", 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781528+0000", 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3799999999999997e-07 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781533+0000", 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6200000000000004e-06 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781539+0000", 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3229999999999997e-06 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781542+0000", 2024-09-13T11:33:44.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3909999999999998e-06 2024-09-13T11:33:44.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.789057+0000", 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075145059999999998 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.789122+0000", 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5436000000000002e-05 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.790113+0000", 2024-09-13T11:33:44.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00099119300000000002 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3870 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~3546310,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.942112550000001, 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012108255, 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.810 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3870 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0999999999999999e-07 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781568+0000", 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.424e-06 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781572+0000", 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6800000000000001e-06 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781576+0000", 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9899999999999999e-06 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.790135+0000", 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085584449999999996 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.790187+0000", 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1683e-05 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.793673+0000", 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034866229999999999 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4151 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3411854,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:33:44.814 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.8290334379999997, 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093747399999999995, 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4151 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:33:44.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9400000000000001e-07 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894647+0000", 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894654+0000", 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894659+0000", 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5600000000000004e-06 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.897162+0000", 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025029829999999999 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.897254+0000", 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1594000000000004e-05 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.904019+0000", 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067652600000000004 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.818 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4152 4.12 4:4d4c8512:::smithi20438004-29:head [sparse-read 0~2953102] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:38.894681+0000", 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 5.8289964300000001, 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011384893, 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.819 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4152 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894681+0000", 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894681+0000", 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894682+0000", 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9399999999999995e-07 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894684+0000", 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4080000000000001e-06 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894689+0000", 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2129999999999999e-06 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894693+0000", 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2050000000000004e-06 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.904055+0000", 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093621360000000001 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.904118+0000", 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2226999999999999e-05 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.906066+0000", 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019482099999999999 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4256 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3411854,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:40.355909+0000", 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.3677687030000003, 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083641099999999993, 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:44.823 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4256 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355909+0000", 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:44.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355909+0000", 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355910+0000", 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6000000000000003e-07 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355917+0000", 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8900000000000007e-06 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355926+0000", 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5359999999999992e-06 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355950+0000", 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3813000000000001e-05 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.359612+0000", 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036622780000000001 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.359678+0000", 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6050999999999995e-05 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.364273+0000", 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045947820000000004 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:44.827 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:44.829 INFO:tasks.rados.rados.0.smithi204.stdout:1819: expect (ObjNum 373 snap 0 seq_num 373) 2024-09-13T11:33:44.829 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1150715 2024-09-13T11:33:44.830 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:44.830 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:44.828+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:44.850 INFO:tasks.rados.rados.0.smithi204.stdout:1820: got (0) Success 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 109 (ObjNum 352 snap 0 seq_num 352) dirty exists 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1814: done (6 left) 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1815: done (5 left) 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1816: done (4 left) 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1817: done (3 left) 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1818: done (2 left) 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1819: done (1 left) 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1820: done (0 left) 2024-09-13T11:33:44.851 INFO:tasks.rados.rados.0.smithi204.stdout:1821: set_chunk oid 35 target oid 2024-09-13T11:33:44.862 INFO:tasks.rados.rados.0.smithi204.stdout:1821: set_chunk oid 35 offset: 2717696 length: 435200 target oid aaf777d39294b0138260685ce65d49ebb973d5c69a2c4329aaa4039201eb3b20 offset: 0 snap -1 2024-09-13T11:33:44.862 INFO:tasks.rados.rados.0.smithi204.stdout:1822: read oid 8 snap -1 2024-09-13T11:33:44.862 INFO:tasks.rados.rados.0.smithi204.stdout:1822: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-13T11:33:44.862 INFO:tasks.rados.rados.0.smithi204.stdout:1823: read oid 37 snap -1 2024-09-13T11:33:44.862 INFO:tasks.rados.rados.0.smithi204.stdout:1823: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:44.862 INFO:tasks.rados.rados.0.smithi204.stdout:1824: write (excl) oid 46 current snap is 0 2024-09-13T11:33:44.863 INFO:tasks.rados.rados.0.smithi204.stdout:1824: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-13T11:33:44.869 INFO:tasks.rados.rados.0.smithi204.stdout:1824: writing smithi20438004-46 from 470695 to 1259210 tid 1 2024-09-13T11:33:44.876 INFO:tasks.rados.rados.0.smithi204.stdout:1824: writing smithi20438004-46 from 1679685 to 2266143 tid 2 2024-09-13T11:33:44.876 INFO:tasks.rados.rados.0.smithi204.stdout:1825: read oid 45 snap -1 2024-09-13T11:33:44.876 INFO:tasks.rados.rados.0.smithi204.stdout:1825: expect (ObjNum 323 snap 0 seq_num 323) 2024-09-13T11:33:44.876 INFO:tasks.rados.rados.0.smithi204.stdout:1826: read oid 32 snap -1 2024-09-13T11:33:44.876 INFO:tasks.rados.rados.0.smithi204.stdout:1826: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:44.876 INFO:tasks.rados.rados.0.smithi204.stdout:1827: write (excl) oid 50 current snap is 0 2024-09-13T11:33:44.876 INFO:tasks.rados.rados.0.smithi204.stdout:1821: finishing set_chunk to oid 35 2024-09-13T11:33:44.877 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 2717696 length: 435200 2024-09-13T11:33:44.877 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 0 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-09-13T11:33:44.877 INFO:tasks.rados.rados.0.smithi204.stdout:1827: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-13T11:33:44.881 INFO:tasks.rados.rados.0.smithi204.stdout:1827: writing smithi20438004-50 from 487502 to 951266 tid 1 2024-09-13T11:33:44.888 INFO:tasks.rados.rados.0.smithi204.stdout:1827: writing smithi20438004-50 from 1738167 to 2184636 tid 2 2024-09-13T11:33:44.889 INFO:tasks.rados.rados.0.smithi204.stdout:1821: done (6 left) 2024-09-13T11:33:44.889 INFO:tasks.rados.rados.0.smithi204.stdout:1828: write (excl) oid 9 current snap is 0 2024-09-13T11:33:44.889 INFO:tasks.rados.rados.0.smithi204.stdout:1828: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-13T11:33:44.894 INFO:tasks.rados.rados.0.smithi204.stdout:1828: writing smithi20438004-9 from 504309 to 1043322 tid 1 2024-09-13T11:33:44.902 INFO:tasks.rados.rados.0.smithi204.stdout:1828: writing smithi20438004-9 from 1796649 to 2386776 tid 2 2024-09-13T11:33:44.902 INFO:tasks.rados.rados.0.smithi204.stdout:1829: write oid 42 current snap is 0 2024-09-13T11:33:44.903 INFO:tasks.rados.rados.0.smithi204.stdout:1829: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-13T11:33:44.909 INFO:tasks.rados.rados.0.smithi204.stdout:1829: writing smithi20438004-42 from 521116 to 1251731 tid 1 2024-09-13T11:33:44.916 INFO:tasks.rados.rados.0.smithi204.stdout:1829: writing smithi20438004-42 from 1971484 to 2521116 tid 2 2024-09-13T11:33:44.916 INFO:tasks.rados.rados.0.smithi204.stdout:1830: read oid 38 snap -1 2024-09-13T11:33:44.916 INFO:tasks.rados.rados.0.smithi204.stdout:1830: expect (ObjNum 351 snap 0 seq_num 351) 2024-09-13T11:33:44.916 INFO:tasks.rados.rados.0.smithi204.stdout:1831: write oid 26 current snap is 0 2024-09-13T11:33:44.916 INFO:tasks.rados.rados.0.smithi204.stdout:1823: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:44.930 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:44.928+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:44.946 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:44.946 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:44.946 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:44.955 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:44.961 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1035141 2024-09-13T11:33:44.976 INFO:tasks.rados.rados.0.smithi204.stdout:1822: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-13T11:33:45.030 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:45.029+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:45.065 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:45.066 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:45.066 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:45.066 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:45.066 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:45.074 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:45.074 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 410042 2024-09-13T11:33:45.131 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:45.129+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:45.222 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:45.222 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:45.223 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:45.223 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:45.223 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.223 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:45.223 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:45.224 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.862471127000006, 2024-09-13T11:33:45.224 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:33:45.224 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.224 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.224 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.225 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.225 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:33:45.225 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:45.225 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:33:45.225 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.226 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.226 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.226 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.226 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:45.226 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.226 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.227 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.227 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:33:45.227 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.227 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.227 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.228 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.228 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:33:45.228 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:33:45.228 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.229 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.229 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.229 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:33:45.230 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:33:45.230 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.230 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.230 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.230 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:33:45.231 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:33:45.231 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.231 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.231 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.231 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:33:45.232 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:33:45.232 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.232 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.232 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.232 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:33:45.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:33:45.233 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.233 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.233 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.233 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:33:45.233 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:33:45.234 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.234 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.234 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.234 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:33:45.234 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:33:45.235 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.235 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.235 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.235 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:33:45.236 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:33:45.236 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.236 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.236 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.236 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.236 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.237 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:45.237 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:45.237 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.113940205999995, 2024-09-13T11:33:45.237 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:33:45.237 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.237 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.238 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.238 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.238 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.238 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.238 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:45.238 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.239 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.240 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.241 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.242 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.101722612000003, 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.243 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:45.244 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.245 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.246 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.247 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1679 4.4 4:2b094039:::smithi20438004-40:head [tier-flush] snapc 0=[] RETRY=3 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:45.248 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.947598505000002, 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0158098280000001, 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1679 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.249 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269187+0000", 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9479999999999998e-06 2024-09-13T11:33:45.250 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269193+0000", 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.173e-06 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269209+0000", 2024-09-13T11:33:45.251 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6039e-05 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269224+0000", 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5494000000000001e-05 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270767+0000", 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015428869999999999 2024-09-13T11:33:45.252 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270782+0000", 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.519e-05 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284573+0000", 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013791077000000001 2024-09-13T11:33:45.253 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.255 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.256 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.256 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284717+0000", 2024-09-13T11:33:45.256 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014369599999999999 2024-09-13T11:33:45.256 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.256 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.257 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.257 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284994+0000", 2024-09-13T11:33:45.257 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.000277324 2024-09-13T11:33:45.257 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.257 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.257 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2310 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.116976960000002, 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17044168200000001, 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.258 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2310 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.259 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.260 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:45.260 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.260 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.260 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.260 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:33:45.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.261 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.261 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.261 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.261 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099808+0000", 2024-09-13T11:33:45.261 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100126+0000", 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031826000000000001 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.262 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100144+0000", 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8029999999999998e-05 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100153+0000", 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0119999999999994e-06 2024-09-13T11:33:45.263 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.264 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100230+0000", 2024-09-13T11:33:45.264 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6221000000000001e-05 2024-09-13T11:33:45.264 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.264 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.264 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100242+0000", 2024-09-13T11:33:45.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.257e-05 2024-09-13T11:33:45.265 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.265 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.265 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.265 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270222+0000", 2024-09-13T11:33:45.265 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16997934100000001 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270248+0000", 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6366000000000001e-05 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.266 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2311 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.110436684, 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16403588, 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.267 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2311 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.268 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:33:45.269 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106347+0000", 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106673+0000", 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588799999999998 2024-09-13T11:33:45.270 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106687+0000", 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3247e-05 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106693+0000", 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2249999999999997e-06 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.271 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106741+0000", 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8715999999999999e-05 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106750+0000", 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3159999999999997e-06 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.272 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270375+0000", 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16362501300000001 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270382+0000", 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.273 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2312 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.105145422, 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158778577, 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2312 2024-09-13T11:33:45.274 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.275 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111639+0000", 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111756+0000", 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117166 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.276 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111770+0000", 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3492999999999999e-05 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111778+0000", 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5820000000000004e-06 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111834+0000", 2024-09-13T11:33:45.277 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6564000000000003e-05 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111842+0000", 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8280000000000003e-06 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270410+0000", 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15856827400000001 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.278 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270416+0000", 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2313 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.104991091000002, 2024-09-13T11:33:45.279 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158679564, 2024-09-13T11:33:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.280 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2313 2024-09-13T11:33:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.281 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.282 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6399999999999998e-07 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.283 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111795+0000", 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0450000000000001e-06 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111799+0000", 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5250000000000001e-06 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.284 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111802+0000", 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.162e-06 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111876+0000", 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4062999999999999e-05 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.285 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111885+0000", 2024-09-13T11:33:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0850000000000003e-06 2024-09-13T11:33:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270465+0000", 2024-09-13T11:33:45.286 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158579839 2024-09-13T11:33:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270472+0000", 2024-09-13T11:33:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4810000000000003e-06 2024-09-13T11:33:45.287 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4.8a282f38 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.104960875000003, 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15868154300000001, 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.288 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:33:45.289 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111823+0000", 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3299999999999998e-07 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111824+0000", 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7290000000000001e-06 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.290 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111829+0000", 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5859999999999998e-06 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111832+0000", 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.456e-06 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.291 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111901+0000", 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8447000000000004e-05 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111905+0000", 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8569999999999997e-06 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.292 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270497+0000", 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158592381 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270504+0000", 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.754e-06 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.293 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2309 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.947465757000003, 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.00703426, 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.294 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2309 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:33:45.295 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269319+0000", 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.265e-06 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269324+0000", 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1800000000000004e-06 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.296 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269334+0000", 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7310000000000005e-06 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269343+0000", 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8339999999999994e-06 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.297 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269381+0000", 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7709000000000001e-05 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269391+0000", 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.298 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278002+0000", 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086114859999999998 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278150+0000", 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014796199999999999 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.276352+0000", 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99820153300000003 2024-09-13T11:33:45.299 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2170 4.12 4:4b088bf8:::smithi20438004-24:head [write 2083958~603055] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.938899894999999, 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029083859, 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.300 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2170 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.301 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277884+0000", 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4399999999999998e-07 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278124+0000", 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000239485 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278133+0000", 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3219999999999999e-06 2024-09-13T11:33:45.302 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278141+0000", 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280144+0000", 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020033289999999999 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280155+0000", 2024-09-13T11:33:45.303 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0966e-05 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301300+0000", 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021145557999999998 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301414+0000", 2024-09-13T11:33:45.304 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011306099999999999 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301601+0000", 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000187357 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304518+0000", 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029169539999999998 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.305 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304561+0000", 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229000000000002e-05 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306715+0000", 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002153706 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306792+0000", 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7466999999999995e-05 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.306 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306851+0000", 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8811999999999999e-05 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306967+0000", 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116062 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.307 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2171 4.12 4:4b088bf8:::smithi20438004-24:head [setxattr _header (56) in=7b,truncate 3058874] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.938627326999999, 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029067939000000001, 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.308 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2171 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.309 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278157+0000", 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0699999999999996e-07 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278161+0000", 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2640000000000002e-06 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.310 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278173+0000", 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1868999999999999e-05 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278179+0000", 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280537+0000", 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023576320000000001 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.311 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280544+0000", 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7560000000000002e-06 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303651+0000", 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023107240000000001 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.312 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303802+0000", 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150774 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303982+0000", 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018009800000000001 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.313 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306699+0000", 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027165100000000001 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306720+0000", 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1535e-05 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306753+0000", 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3019999999999999e-05 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307010+0000", 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025724800000000001 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307061+0000", 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0633000000000001e-05 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307224+0000", 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000162843 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2172 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.938594643000002, 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029398245999999999, 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.316 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2172 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000005e-07 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278193+0000", 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278201+0000", 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278207+0000", 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4289999999999998e-06 2024-09-13T11:33:45.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280568+0000", 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023610900000000002 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280577+0000", 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8259999999999999e-06 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304605+0000", 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024027829000000001 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304704+0000", 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9610000000000003e-05 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307334+0000", 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002629667 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307421+0000", 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7433000000000001e-05 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307587+0000", 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016528800000000001 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2219 4.12 4:4d4c8512:::smithi20438004-29:head [sparse-read 0~2348017] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.322 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.938523435, 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029874108999999999, 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2219 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3599999999999999e-07 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278262+0000", 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2110000000000001e-06 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278269+0000", 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.314e-06 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278274+0000", 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5249999999999996e-06 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280850+0000", 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025761389999999999 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280858+0000", 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8760000000000005e-06 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307628+0000", 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.026769465999999999 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307703+0000", 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4847999999999999e-05 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.308134+0000", 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00043129399999999998 2024-09-13T11:33:45.327 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.936767967999998, 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029229465, 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.328 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280016+0000", 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280020+0000", 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280026+0000", 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1070000000000001e-06 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280032+0000", 2024-09-13T11:33:45.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280109+0000", 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6298999999999996e-05 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280116+0000", 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5129999999999999e-06 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298271+0000", 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018154805 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298334+0000", 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3268999999999993e-05 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309030+0000", 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010695566 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309089+0000", 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309245+0000", 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000155317 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2340 4.18 4:1d367c64:::smithi20438004-45:head [write 578207~550227] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.334 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.936740221000001, 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030475704999999999, 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2340 2024-09-13T11:33:45.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.336 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0399999999999996e-07 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280452+0000", 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040812800000000001 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280461+0000", 2024-09-13T11:33:45.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3829999999999998e-06 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280468+0000", 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0400000000000004e-06 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280506+0000", 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8037000000000002e-05 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280514+0000", 2024-09-13T11:33:45.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.011e-06 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298673+0000", 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018159411 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298829+0000", 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015594 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:45.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.299023+0000", 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193838 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307549+0000", 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085256289999999998 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307635+0000", 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6596999999999998e-05 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307689+0000", 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4027999999999998e-05 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307748+0000", 2024-09-13T11:33:45.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8733999999999997e-05 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310255+0000", 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025070940000000001 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310375+0000", 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119691 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310519+0000", 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014364 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2341 4.18 4:1d367c64:::smithi20438004-45:head [write 1762410~779734] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.936297854000003, 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031687067999999999, 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.343 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2341 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280486+0000", 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281018+0000", 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053231899999999996 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281027+0000", 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.704e-06 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281034+0000", 2024-09-13T11:33:45.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0659999999999998e-06 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281159+0000", 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124661 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281168+0000", 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000001e-06 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302725+0000", 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021557478000000001 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302889+0000", 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016421299999999999 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303087+0000", 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019817200000000001 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307806+0000", 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004718168 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307868+0000", 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2775999999999997e-05 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307920+0000", 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1749000000000002e-05 2024-09-13T11:33:45.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307971+0000", 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1343e-05 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311829+0000", 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038571769999999998 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312043+0000", 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021412199999999999 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312172+0000", 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012966900000000001 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.351 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2342 4.18 4:1d367c64:::smithi20438004-45:head [write 3286845~91362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.935730702999997, 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031505618999999999, 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.352 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2342 2024-09-13T11:33:45.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.353 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:45.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281053+0000", 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2799999999999995e-07 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281104+0000", 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0550999999999998e-05 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281112+0000", 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0129999999999993e-06 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281118+0000", 2024-09-13T11:33:45.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281191+0000", 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2738999999999996e-05 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281197+0000", 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309351+0000", 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028153293999999999 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309441+0000", 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0048000000000002e-05 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309594+0000", 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153609 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311315+0000", 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017209829999999999 2024-09-13T11:33:45.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311795+0000", 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048009300000000001 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312281+0000", 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048612699999999999 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.359 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312394+0000", 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112935 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312404+0000", 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0112000000000001e-05 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312449+0000", 2024-09-13T11:33:45.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4650000000000001e-05 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312558+0000", 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108959 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2343 4.18 4:1d367c64:::smithi20438004-45:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.935655152999999, 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031723945000000003, 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2343 2024-09-13T11:33:45.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281129+0000", 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281132+0000", 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.988e-06 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281137+0000", 2024-09-13T11:33:45.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.95e-06 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281142+0000", 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4e-06 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281219+0000", 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6946000000000005e-05 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281225+0000", 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.601e-06 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310552+0000", 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029326497 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310643+0000", 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1416999999999996e-05 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310817+0000", 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000173463 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312229+0000", 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001412424 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312324+0000", 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4264000000000004e-05 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312578+0000", 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025401699999999999 2024-09-13T11:33:45.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312605+0000", 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7557000000000001e-05 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312650+0000", 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4496000000000002e-05 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312683+0000", 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2898000000000002e-05 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312852+0000", 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000169436 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2344 4.18 4:1d367c64:::smithi20438004-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 47.935635406000003, 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032021714999999999, 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.372 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.372 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.372 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2344 2024-09-13T11:33:45.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.372 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:45.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:45.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:33:45.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.15e-07 2024-09-13T11:33:45.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281150+0000", 2024-09-13T11:33:45.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.857e-06 2024-09-13T11:33:45.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281155+0000", 2024-09-13T11:33:45.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0579999999999997e-06 2024-09-13T11:33:45.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281160+0000", 2024-09-13T11:33:45.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0409999999999997e-06 2024-09-13T11:33:45.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.378 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.378 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.378 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281248+0000", 2024-09-13T11:33:45.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7979000000000006e-05 2024-09-13T11:33:45.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281255+0000", 2024-09-13T11:33:45.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5089999999999999e-06 2024-09-13T11:33:45.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311169+0000", 2024-09-13T11:33:45.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02991394 2024-09-13T11:33:45.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311266+0000", 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7007000000000001e-05 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312894+0000", 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016280489999999999 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312976+0000", 2024-09-13T11:33:45.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2052999999999995e-05 2024-09-13T11:33:45.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.313169+0000", 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193807 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.383 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.384 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:45.386 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:45.388 INFO:tasks.rados.rados.0.smithi204.stdout:1824: finishing write tid 1 to smithi20438004-46 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1824: finishing write tid 2 to smithi20438004-46 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1824: finishing write tid 3 to smithi20438004-46 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1824: finishing write tid 4 to smithi20438004-46 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1824: oid 46 updating version 0 to 104 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1824: oid 46 updating version 104 to 106 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1824: oid 46 version 106 is already newer than 105 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 46 v 106 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1824: left oid 46 (ObjNum 385 snap 0 seq_num 385) 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1831: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1831: writing smithi20438004-26 from 537923 to 943787 tid 1 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1831: writing smithi20438004-26 from 1513613 to 2107409 tid 2 2024-09-13T11:33:45.389 INFO:tasks.rados.rados.0.smithi204.stdout:1822: done (9 left) 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1823: done (8 left) 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1824: done (7 left) 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1832: write oid 14 current snap is 0 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1832: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1832: writing smithi20438004-14 from 554730 to 1035843 tid 1 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1832: writing smithi20438004-14 from 1572095 to 2309549 tid 2 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1833: write (excl) oid 27 current snap is 0 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1833: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1833: writing smithi20438004-27 from 571537 to 1127899 tid 1 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1833: writing smithi20438004-27 from 1630577 to 2228042 tid 2 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1834: read oid 31 snap -1 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1834: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:45.390 INFO:tasks.rados.rados.0.smithi204.stdout:1835: write oid 34 current snap is 0 2024-09-13T11:33:45.391 INFO:tasks.rados.rados.0.smithi204.stdout:1825: expect (ObjNum 323 snap 0 seq_num 323) 2024-09-13T11:33:45.391 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:45.391 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:45.384+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:45.486 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:45.485+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:45.500 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:45.501 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:45.501 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:45.501 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:45.510 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:45.510 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2630032 2024-09-13T11:33:45.523 INFO:tasks.rados.rados.0.smithi204.stdout:1826: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:45.523 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 18326 2024-09-13T11:33:45.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:45.587 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:45.586+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:45.614 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:45.614 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:45.615 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:45.615 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:45.615 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:45.623 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:45.625 INFO:tasks.rados.rados.0.smithi204.stdout:1827: finishing write tid 1 to smithi20438004-50 2024-09-13T11:33:45.625 INFO:tasks.rados.rados.0.smithi204.stdout:1827: finishing write tid 2 to smithi20438004-50 2024-09-13T11:33:45.625 INFO:tasks.rados.rados.0.smithi204.stdout:1835: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-13T11:33:45.632 INFO:tasks.rados.rados.0.smithi204.stdout:1835: writing smithi20438004-34 from 588344 to 1219955 tid 1 2024-09-13T11:33:45.639 INFO:tasks.rados.rados.0.smithi204.stdout:1835: writing smithi20438004-34 from 1689059 to 2430182 tid 2 2024-09-13T11:33:45.639 INFO:tasks.rados.rados.0.smithi204.stdout:1825: done (10 left) 2024-09-13T11:33:45.639 INFO:tasks.rados.rados.0.smithi204.stdout:1826: done (9 left) 2024-09-13T11:33:45.639 INFO:tasks.rados.rados.0.smithi204.stdout:1836: read oid 7 snap -1 2024-09-13T11:33:45.639 INFO:tasks.rados.rados.0.smithi204.stdout:1836: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-13T11:33:45.639 INFO:tasks.rados.rados.0.smithi204.stdout:1837: set_chunk oid 20 target oid 2024-09-13T11:33:45.688 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:45.687+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:45.691 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:45.691 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:45.709 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:45.709 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:45.710 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:45.726 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:45.727 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:45.733 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:45.733 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:45.733 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:45.733 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.633875441000001, 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.734 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.735 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:33:45.736 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:33:45.737 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:33:45.738 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:33:45.739 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.739 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.739 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.739 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:33:45.739 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:33:45.739 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.739 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.621441296, 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.740 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.741 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:33:45.742 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.743 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.744 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.611487702000005, 2024-09-13T11:33:45.745 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:45.746 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.747 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.748 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:33:45.749 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.750 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:33:45.751 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.752 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.752 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.752 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:33:45.752 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:33:45.752 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.752 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.752 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.580769895000003, 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.753 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.754 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.754 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.754 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:45.754 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.754 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.754 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:33:45.755 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.756 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.757 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:33:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:33:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.758 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.759 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.760 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.561946575999997, 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.761 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:45.762 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:33:45.763 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.764 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.765 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.766 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.766 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:33:45.766 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:33:45.766 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.766 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.766 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.766 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.767 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.767 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:45.767 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:45.767 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.548427227000005, 2024-09-13T11:33:45.767 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:33:45.767 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.767 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.768 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.768 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.768 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.768 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.768 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:45.768 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.768 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.769 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.770 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:33:45.771 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.772 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.538786893999998, 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.774 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:33:45.775 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.776 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:33:45.777 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.528481393999996, 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.778 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.779 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.780 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:33:45.781 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.782 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.783 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.518095451999997, 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.784 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:33:45.785 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.786 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.787 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.788 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:36 3.3 3:e2816a99:::d4bb749789c90c884ffd20ac62f804c7a6b80502d14432fa627e1239249ee2da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.460568053000003, 2024-09-13T11:33:45.789 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013305613000000001, 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.790 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270480+0000", 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2399999999999998e-07 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.791 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270655+0000", 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017481300000000001 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270683+0000", 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8340999999999999e-05 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.792 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270693+0000", 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271029+0000", 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033640200000000002 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.793 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271038+0000", 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278355+0000", 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073166510000000004 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278481+0000", 2024-09-13T11:33:45.794 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125955 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278657+0000", 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176816 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280615+0000", 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019576810000000002 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.795 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283538+0000", 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029229030000000001 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283592+0000", 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4366000000000001e-05 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.796 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283671+0000", 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8337000000000006e-05 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283785+0000", 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114412 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.797 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:40 3.3 3:c7a12847:::768a08ae37e9bf19369a517906ebd209c1f2b8ad3684596d644da7d48b42114d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.460324946999997, 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013347114, 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.798 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.799 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270839+0000", 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115369 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270847+0000", 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.800 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270854+0000", 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271073+0000", 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021964400000000001 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271080+0000", 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7909999999999999e-06 2024-09-13T11:33:45.801 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278928+0000", 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078479819999999999 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.802 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279058+0000", 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130102 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279300+0000", 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024166300000000001 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280641+0000", 2024-09-13T11:33:45.803 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013409229999999999 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283545+0000", 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002904401 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283927+0000", 2024-09-13T11:33:45.804 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038193199999999997 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283964+0000", 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7117999999999999e-05 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284070+0000", 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010545899999999999 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.805 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:41 3.3 3:e9fa9a15:::3c65c6c9770b7a29cd5a9b6dc5b0b39ab23cb2a64bfb19442efcb911d255e067:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.460181333999998, 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013499547000000001, 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.806 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.807 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270867+0000", 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270893+0000", 2024-09-13T11:33:45.808 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6057000000000001e-05 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270901+0000", 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270915+0000", 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4450000000000001e-05 2024-09-13T11:33:45.809 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271104+0000", 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018904600000000001 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271111+0000", 2024-09-13T11:33:45.810 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6039999999999998e-06 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280674+0000", 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095629140000000005 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280811+0000", 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136802 2024-09-13T11:33:45.811 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281027+0000", 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000216617 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283802+0000", 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027746540000000001 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.812 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283884+0000", 2024-09-13T11:33:45.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2167000000000001e-05 2024-09-13T11:33:45.813 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.813 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.813 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.813 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284178+0000", 2024-09-13T11:33:45.813 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029347899999999997 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284246+0000", 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8609999999999995e-05 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284366+0000", 2024-09-13T11:33:45.814 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011950799999999999 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:43 3.3 3:dc8816b1:::1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 59.460120639000003, 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013665096999999999, 2024-09-13T11:33:45.815 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.816 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-13T11:33:45.817 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270933+0000", 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2410000000000003e-06 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270939+0000", 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5810000000000002e-06 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270945+0000", 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271134+0000", 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000189161 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271140+0000", 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2999999999999998e-06 2024-09-13T11:33:45.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281407+0000", 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010266226 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281549+0000", 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142583 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:45.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281737+0000", 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018791500000000001 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283810+0000", 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002072571 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284127+0000", 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000317516 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284417+0000", 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029000899999999998 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284480+0000", 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3136e-05 2024-09-13T11:33:45.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284592+0000", 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111712 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.823 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.886292767, 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43316489499999999, 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.824 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844756+0000", 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844760+0000", 2024-09-13T11:33:45.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0550000000000001e-06 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844769+0000", 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6129999999999993e-06 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844776+0000", 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0280000000000003e-06 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7639000000000002e-05 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844841+0000", 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1049999999999997e-06 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277896+0000", 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43305493499999997 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277920+0000", 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3614000000000001e-05 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.886241134000002, 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43315226600000001, 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.830 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844807+0000", 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3200000000000003e-07 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844811+0000", 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.799e-06 2024-09-13T11:33:45.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844827+0000", 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5461999999999999e-05 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.126e-06 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5369999999999999e-05 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844863+0000", 2024-09-13T11:33:45.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.658e-06 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277951+0000", 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308847900000003 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277959+0000", 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-13T11:33:45.835 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 49.886199423000001, 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43314228399999999, 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.836 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844849+0000", 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0100000000000005e-07 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844853+0000", 2024-09-13T11:33:45.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9589999999999997e-06 2024-09-13T11:33:45.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4110000000000001e-06 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844864+0000", 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8930000000000004e-06 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844892+0000", 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8042e-05 2024-09-13T11:33:45.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844898+0000", 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5049999999999998e-06 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277984+0000", 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308647700000003 2024-09-13T11:33:45.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277990+0000", 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3960000000000003e-06 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4:8c30e60e:::smithi20438004-18:head [read 0~2487436,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.448650100999998, 2024-09-13T11:33:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012791258999999999, 2024-09-13T11:33:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.843 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282397+0000", 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282399+0000", 2024-09-13T11:33:45.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.691e-06 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282404+0000", 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.887e-06 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282414+0000", 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9539999999999999e-06 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282423+0000", 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6039999999999993e-06 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282477+0000", 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4738999999999999e-05 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282486+0000", 2024-09-13T11:33:45.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.5450000000000003e-06 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287493+0000", 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0050068949999999999 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.287636+0000", 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014333299999999999 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.295189+0000", 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075526109999999999 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 153896~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:45.849 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.448548510999998, 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014013433, 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282500+0000", 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5499999999999998e-07 2024-09-13T11:33:45.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282503+0000", 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.563e-06 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282510+0000", 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8820000000000003e-06 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282516+0000", 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282562+0000", 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5868999999999997e-05 2024-09-13T11:33:45.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282570+0000", 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296085+0000", 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013515255 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296187+0000", 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101457 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296512+0000", 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588099999999998 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 447518~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:45.856 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 48.448464018000003, 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014496826000000001, 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5000000000000002e-07 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282589+0000", 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4220000000000002e-06 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282595+0000", 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8789999999999997e-06 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282606+0000", 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0229e-05 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282652+0000", 2024-09-13T11:33:45.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6610999999999998e-05 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282661+0000", 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2640000000000008e-06 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296550+0000", 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013888981999999999 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296628+0000", 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8564000000000002e-05 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.297080+0000", 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000452225 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3902 4.11 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:31.072365+0000", 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 14.658682280000001, 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011701236, 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:45.863 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3902 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072365+0000", 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072365+0000", 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072367+0000", 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.668e-06 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072392+0000", 2024-09-13T11:33:45.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5358e-05 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072407+0000", 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4669e-05 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072415+0000", 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8439999999999998e-06 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072475+0000", 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0424000000000003e-05 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.072568+0000", 2024-09-13T11:33:45.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3255000000000001e-05 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:31.084066+0000", 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011498018 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:45.868 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:45.869 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:45.870 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:45.870 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:45.871 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:45.871 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:45.872 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:45.872 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:45.872 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:45.873 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:45.873 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:45.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:45.874 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:45.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:45.875 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:45.875 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:45.876 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:45.876 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:45.876 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:45.868+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:45.877 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:45.877 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:45.892 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:45.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:45.909 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:45.909 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:45.912 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:45.971 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:45.970+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:46.017 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.017 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:46.025 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:46.110 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:46.109+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:46.168 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.168 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:46.173 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:46.171+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:46.178 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:46.274 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:46.273+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:46.321 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.321 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:46.330 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:46.374 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:46.373+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:46.433 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.433 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:46.441 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:46.476 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:46.475+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:46.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:33:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:1837: set_chunk oid 20 offset: 1200640 length: 616448 target oid 38cf479888295858b40d3b83afb7a45eff19b3703a4a52079bce93c7ea725a6b offset: 0 snap -1 2024-09-13T11:33:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:1838: write (excl) oid 18 current snap is 0 2024-09-13T11:33:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:1827: finishing write tid 3 to smithi20438004-50 2024-09-13T11:33:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:1827: finishing write tid 4 to smithi20438004-50 2024-09-13T11:33:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:1827: oid 50 updating version 0 to 77 2024-09-13T11:33:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:1827: oid 50 updating version 77 to 78 2024-09-13T11:33:46.513 INFO:tasks.rados.rados.0.smithi204.stdout:1827: oid 50 updating version 78 to 79 2024-09-13T11:33:46.514 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 79 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-09-13T11:33:46.514 INFO:tasks.rados.rados.0.smithi204.stdout:1827: left oid 50 (ObjNum 386 snap 0 seq_num 386) 2024-09-13T11:33:46.514 INFO:tasks.rados.rados.0.smithi204.stdout:1828: finishing write tid 1 to smithi20438004-9 2024-09-13T11:33:46.514 INFO:tasks.rados.rados.0.smithi204.stdout:1828: finishing write tid 2 to smithi20438004-9 2024-09-13T11:33:46.514 INFO:tasks.rados.rados.0.smithi204.stdout:1828: finishing write tid 3 to smithi20438004-9 2024-09-13T11:33:46.514 INFO:tasks.rados.rados.0.smithi204.stdout:1838: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-13T11:33:46.520 INFO:tasks.rados.rados.0.smithi204.stdout:1838: writing smithi20438004-18 from 605151 to 1312011 tid 1 2024-09-13T11:33:46.527 INFO:tasks.rados.rados.0.smithi204.stdout:1838: writing smithi20438004-18 from 2031188 to 2605151 tid 2 2024-09-13T11:33:46.527 INFO:tasks.rados.rados.0.smithi204.stdout:1827: done (11 left) 2024-09-13T11:33:46.527 INFO:tasks.rados.rados.0.smithi204.stdout:1839: set_chunk oid 45 target oid 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1839: set_chunk oid 45 offset: 882176 length: 1519616 target oid a41bbc3a9f2bc6473f4d617591367f87ac24a07d0067b45551636639d17f8997 offset: 0 snap -1 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1840: read oid 47 snap -1 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1840: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1841: read oid 33 snap -1 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1841: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1842: read oid 22 snap -1 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1842: expect (ObjNum 352 snap 0 seq_num 352) 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1843: write (excl) oid 23 current snap is 0 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1828: finishing write tid 4 to smithi20438004-9 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1828: oid 9 updating version 0 to 67 2024-09-13T11:33:46.541 INFO:tasks.rados.rados.0.smithi204.stdout:1828: oid 9 updating version 67 to 68 2024-09-13T11:33:46.542 INFO:tasks.rados.rados.0.smithi204.stdout:1828: oid 9 updating version 68 to 69 2024-09-13T11:33:46.542 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 69 (ObjNum 387 snap 0 seq_num 387) dirty exists 2024-09-13T11:33:46.542 INFO:tasks.rados.rados.0.smithi204.stdout:1828: left oid 9 (ObjNum 387 snap 0 seq_num 387) 2024-09-13T11:33:46.542 INFO:tasks.rados.rados.0.smithi204.stdout:1829: finishing write tid 1 to smithi20438004-42 2024-09-13T11:33:46.542 INFO:tasks.rados.rados.0.smithi204.stdout:1843: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-13T11:33:46.549 INFO:tasks.rados.rados.0.smithi204.stdout:1843: writing smithi20438004-23 from 621958 to 1404067 tid 1 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1843: writing smithi20438004-23 from 2089670 to 2621958 tid 2 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1828: done (15 left) 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1844: write (excl) oid 50 current snap is 0 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1829: finishing write tid 2 to smithi20438004-42 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1829: finishing write tid 3 to smithi20438004-42 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1829: finishing write tid 4 to smithi20438004-42 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1829: oid 42 updating version 0 to 105 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1829: oid 42 version 105 is already newer than 103 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1829: oid 42 version 105 is already newer than 104 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 105 (ObjNum 388 snap 0 seq_num 388) dirty exists 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1829: left oid 42 (ObjNum 388 snap 0 seq_num 388) 2024-09-13T11:33:46.557 INFO:tasks.rados.rados.0.smithi204.stdout:1830: expect (ObjNum 351 snap 0 seq_num 351) 2024-09-13T11:33:46.577 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:46.576+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:46.603 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.603 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:46.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:46.695 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:46.694+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:46.722 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:46.756 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.756 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:46.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:46.778 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:46.777+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: finishing write tid 1 to smithi20438004-26 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: finishing write tid 2 to smithi20438004-26 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: finishing write tid 3 to smithi20438004-26 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: finishing write tid 4 to smithi20438004-26 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: oid 26 updating version 0 to 75 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: oid 26 updating version 75 to 76 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: oid 26 updating version 76 to 77 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 77 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1831: left oid 26 (ObjNum 389 snap 0 seq_num 389) 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1832: finishing write tid 1 to smithi20438004-14 2024-09-13T11:33:46.812 INFO:tasks.rados.rados.0.smithi204.stdout:1832: finishing write tid 2 to smithi20438004-14 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1832: finishing write tid 3 to smithi20438004-14 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1832: finishing write tid 4 to smithi20438004-14 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1832: oid 14 updating version 0 to 136 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1832: oid 14 updating version 136 to 138 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1832: oid 14 version 138 is already newer than 137 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 138 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1832: left oid 14 (ObjNum 390 snap 0 seq_num 390) 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1833: finishing write tid 1 to smithi20438004-27 2024-09-13T11:33:46.813 INFO:tasks.rados.rados.0.smithi204.stdout:1844: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-13T11:33:46.817 INFO:tasks.rados.rados.0.smithi204.stdout:1844: writing smithi20438004-50 from 638765 to 1096123 tid 1 2024-09-13T11:33:46.824 INFO:tasks.rados.rados.0.smithi204.stdout:1844: writing smithi20438004-50 from 1748152 to 2352955 tid 2 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1829: done (15 left) 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1830: done (14 left) 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1831: done (13 left) 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1832: done (12 left) 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1845: read oid 10 snap -1 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1845: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1846: read oid 17 snap -1 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1846: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1847: read oid 13 snap -1 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1847: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1848: write (excl) oid 42 current snap is 0 2024-09-13T11:33:46.825 INFO:tasks.rados.rados.0.smithi204.stdout:1834: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:46.874 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.875 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:46.879 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:46.878+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:46.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:46.894 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1094023 2024-09-13T11:33:46.912 INFO:tasks.rados.rados.0.smithi204.stdout:1833: finishing write tid 2 to smithi20438004-27 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1833: finishing write tid 3 to smithi20438004-27 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1833: finishing write tid 4 to smithi20438004-27 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1833: oid 27 updating version 0 to 134 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1833: oid 27 version 134 is already newer than 133 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1833: oid 27 version 134 is already newer than 132 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 134 (ObjNum 391 snap 0 seq_num 391) dirty exists 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1833: left oid 27 (ObjNum 391 snap 0 seq_num 391) 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1835: finishing write tid 1 to smithi20438004-34 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1835: finishing write tid 2 to smithi20438004-34 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1835: finishing write tid 3 to smithi20438004-34 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1835: finishing write tid 4 to smithi20438004-34 2024-09-13T11:33:46.913 INFO:tasks.rados.rados.0.smithi204.stdout:1835: oid 34 updating version 0 to 148 2024-09-13T11:33:46.914 INFO:tasks.rados.rados.0.smithi204.stdout:1835: oid 34 updating version 148 to 149 2024-09-13T11:33:46.914 INFO:tasks.rados.rados.0.smithi204.stdout:1835: oid 34 version 149 is already newer than 147 2024-09-13T11:33:46.914 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 149 (ObjNum 392 snap 0 seq_num 392) dirty exists 2024-09-13T11:33:46.914 INFO:tasks.rados.rados.0.smithi204.stdout:1835: left oid 34 (ObjNum 392 snap 0 seq_num 392) 2024-09-13T11:33:46.914 INFO:tasks.rados.rados.0.smithi204.stdout:1836: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-13T11:33:46.980 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:46.979+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:46.991 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.000 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:47.018 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2833457 2024-09-13T11:33:47.081 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:47.079+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:47.101 INFO:tasks.rados.rados.0.smithi204.stdout:1837: finishing set_chunk to oid 20 2024-09-13T11:33:47.101 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 101 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-13T11:33:47.101 INFO:tasks.rados.rados.0.smithi204.stdout:1838: finishing write tid 1 to smithi20438004-18 2024-09-13T11:33:47.101 INFO:tasks.rados.rados.0.smithi204.stdout:1838: finishing write tid 2 to smithi20438004-18 2024-09-13T11:33:47.101 INFO:tasks.rados.rados.0.smithi204.stdout:1838: finishing write tid 3 to smithi20438004-18 2024-09-13T11:33:47.101 INFO:tasks.rados.rados.0.smithi204.stdout:1848: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-13T11:33:47.106 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:47.107 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.108 INFO:tasks.rados.rados.0.smithi204.stdout:1848: writing smithi20438004-42 from 655572 to 1304532 tid 1 2024-09-13T11:33:47.116 INFO:tasks.rados.rados.0.smithi204.stdout:1848: writing smithi20438004-42 from 1922987 to 2655572 tid 2 2024-09-13T11:33:47.116 INFO:tasks.rados.rados.0.smithi204.stdout:1833: done (15 left) 2024-09-13T11:33:47.116 INFO:tasks.rados.rados.0.smithi204.stdout:1834: done (14 left) 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1835: done (13 left) 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1836: done (12 left) 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1837: done (11 left) 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1849: write (excl) oid 6 current snap is 0 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1838: finishing write tid 4 to smithi20438004-18 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1838: oid 18 updating version 0 to 150 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1838: oid 18 updating version 150 to 151 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1838: oid 18 updating version 151 to 152 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 152 (ObjNum 393 snap 0 seq_num 393) dirty exists 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1838: left oid 18 (ObjNum 393 snap 0 seq_num 393) 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1839: finishing set_chunk to oid 45 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 882176 length: 1519616 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 0 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-09-13T11:33:47.117 INFO:tasks.rados.rados.0.smithi204.stdout:1840: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-13T11:33:47.118 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:47.153 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:47.181 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:47.179+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:47.190 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:47.226 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:47.226 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.227 INFO:tasks.rados.rados.0.smithi204.stdout:1841: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-13T11:33:47.233 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 507030 2024-09-13T11:33:47.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:47.283 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:47.282+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:47.301 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2370072 2024-09-13T11:33:47.379 INFO:tasks.rados.rados.0.smithi204.stdout:1842: expect (ObjNum 352 snap 0 seq_num 352) 2024-09-13T11:33:47.380 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:47.382 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:47.382 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.384 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:47.383+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:47.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:47.461 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:47.480 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:33:47.484 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:47.482+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:47.504 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:47.504 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:47.589 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:47.584+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:1843: finishing write tid 1 to smithi20438004-23 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:1843: finishing write tid 2 to smithi20438004-23 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:1843: finishing write tid 3 to smithi20438004-23 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:1843: finishing write tid 4 to smithi20438004-23 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:1843: oid 23 updating version 0 to 96 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:1843: oid 23 version 96 is already newer than 95 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:1843: oid 23 updating version 96 to 97 2024-09-13T11:33:47.627 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 97 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-13T11:33:47.628 INFO:tasks.rados.rados.0.smithi204.stdout:1843: left oid 23 (ObjNum 394 snap 0 seq_num 394) 2024-09-13T11:33:47.628 INFO:tasks.rados.rados.0.smithi204.stdout:1849: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-13T11:33:47.634 INFO:tasks.rados.rados.0.smithi204.stdout:1849: writing smithi20438004-6 from 672379 to 1396588 tid 1 2024-09-13T11:33:47.640 INFO:tasks.rados.rados.0.smithi204.stdout:1849: writing smithi20438004-6 from 1865116 to 2473588 tid 2 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1838: done (11 left) 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1839: done (10 left) 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1840: done (9 left) 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1841: done (8 left) 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1842: done (7 left) 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1843: done (6 left) 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1850: write (excl) oid 26 current snap is 0 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1844: finishing write tid 1 to smithi20438004-50 2024-09-13T11:33:47.641 INFO:tasks.rados.rados.0.smithi204.stdout:1846: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-13T11:33:47.662 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:47.662 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:47.686 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:47.684+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:47.710 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 994214 2024-09-13T11:33:47.778 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:47.778 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:47.788 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:47.786+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1844: finishing write tid 2 to smithi20438004-50 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1844: finishing write tid 3 to smithi20438004-50 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1844: finishing write tid 4 to smithi20438004-50 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1844: oid 50 updating version 0 to 82 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1844: oid 50 version 82 is already newer than 80 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1844: oid 50 version 82 is already newer than 81 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 82 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1844: left oid 50 (ObjNum 395 snap 0 seq_num 395) 2024-09-13T11:33:47.822 INFO:tasks.rados.rados.0.smithi204.stdout:1845: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-13T11:33:47.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:47.888 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:47.886+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:47.894 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:47.895 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:47.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:47.911 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1880554 2024-09-13T11:33:47.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:47.920 INFO:tasks.rados.rados.0.smithi204.stdout:1847: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:47.989 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:47.988+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:47.991 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:48.018 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:48.018 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:48.027 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:48.069 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:48.069 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:48.086 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:48.087 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:48.087 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:48.090 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:48.088+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:48.105 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:48.105 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:48.121 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:48.121 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:48.131 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:48.131 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:48.131 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:48.132 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:48.132 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:48.137 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:48.137 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:48.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:48.144 INFO:tasks.rados.rados.0.smithi204.stdout:1848: finishing write tid 1 to smithi20438004-42 2024-09-13T11:33:48.144 INFO:tasks.rados.rados.0.smithi204.stdout:1848: finishing write tid 2 to smithi20438004-42 2024-09-13T11:33:48.144 INFO:tasks.rados.rados.0.smithi204.stdout:1848: finishing write tid 3 to smithi20438004-42 2024-09-13T11:33:48.144 INFO:tasks.rados.rados.0.smithi204.stdout:1848: finishing write tid 4 to smithi20438004-42 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1848: oid 42 updating version 0 to 108 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1848: oid 42 updating version 108 to 109 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1848: oid 42 version 109 is already newer than 107 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 109 (ObjNum 396 snap 0 seq_num 396) dirty exists 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1848: left oid 42 (ObjNum 396 snap 0 seq_num 396) 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: finishing write tid 1 to smithi20438004-6 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: finishing write tid 2 to smithi20438004-6 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: finishing write tid 3 to smithi20438004-6 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: finishing write tid 4 to smithi20438004-6 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: oid 6 updating version 0 to 108 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: oid 6 version 108 is already newer than 107 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: oid 6 updating version 108 to 109 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 109 (ObjNum 397 snap 0 seq_num 397) dirty exists 2024-09-13T11:33:48.145 INFO:tasks.rados.rados.0.smithi204.stdout:1849: left oid 6 (ObjNum 397 snap 0 seq_num 397) 2024-09-13T11:33:48.146 INFO:tasks.rados.rados.0.smithi204.stdout:1850: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-13T11:33:48.151 INFO:tasks.rados.rados.0.smithi204.stdout:1850: writing smithi20438004-26 from 689186 to 1488644 tid 1 2024-09-13T11:33:48.155 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:48.156 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:48.159 INFO:tasks.rados.rados.0.smithi204.stdout:1850: writing smithi20438004-26 from 1923598 to 2675728 tid 2 2024-09-13T11:33:48.159 INFO:tasks.rados.rados.0.smithi204.stdout:1844: done (6 left) 2024-09-13T11:33:48.159 INFO:tasks.rados.rados.0.smithi204.stdout:1845: done (5 left) 2024-09-13T11:33:48.160 INFO:tasks.rados.rados.0.smithi204.stdout:1846: done (4 left) 2024-09-13T11:33:48.160 INFO:tasks.rados.rados.0.smithi204.stdout:1847: done (3 left) 2024-09-13T11:33:48.160 INFO:tasks.rados.rados.0.smithi204.stdout:1848: done (2 left) 2024-09-13T11:33:48.160 INFO:tasks.rados.rados.0.smithi204.stdout:1849: done (1 left) 2024-09-13T11:33:48.160 INFO:tasks.rados.rados.0.smithi204.stdout:1851: set_chunk oid 49 target oid 2024-09-13T11:33:48.171 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:48.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:48.188 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:48.188 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:48.191 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:48.189+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:48.204 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:48.204 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:48.222 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:48.223 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:48.240 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:48.240 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:48.240 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:48.258 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:48.259 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:48.276 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:48.289 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:48.290 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:48.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:48.290 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:48.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.291 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:48.292 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:48.292 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.192503102000003, 2024-09-13T11:33:48.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011006991000000001, 2024-09-13T11:33:48.292 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.292 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.293 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.293 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:48.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:48.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091279+0000", 2024-09-13T11:33:48.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.091e-06 2024-09-13T11:33:48.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098180+0000", 2024-09-13T11:33:48.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069015359999999998 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098194+0000", 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.323e-05 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098202+0000", 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2879999999999993e-06 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098267+0000", 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4733999999999994e-05 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098296+0000", 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102225+0000", 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039281940000000003 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102285+0000", 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.135697613000005, 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010863593, 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:48.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148085+0000", 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2810000000000001e-06 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153254+0000", 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051698430000000004 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153266+0000", 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1973e-05 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153274+0000", 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:33:48.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153331+0000", 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6492000000000001e-05 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153396+0000", 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5083e-05 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158906+0000", 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055099570000000002 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158947+0000", 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:48.305 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.130485066999995, 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096623879999999992, 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157238+0000", 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039414139999999999 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157253+0000", 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5566e-05 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157260+0000", 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.477e-06 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157315+0000", 2024-09-13T11:33:48.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4857e-05 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157341+0000", 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162899+0000", 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055583480000000003 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162958+0000", 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9169000000000002e-05 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.120826981999997, 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011410943999999999, 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.310 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162955+0000", 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0669999999999999e-06 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167312+0000", 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004357072 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167333+0000", 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0842000000000001e-05 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167341+0000", 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4430000000000003e-06 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167406+0000", 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4245000000000006e-05 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167433+0000", 2024-09-13T11:33:48.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7500000000000001e-05 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174280+0000", 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068473969999999999 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174365+0000", 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4377999999999997e-05 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.107999219000007, 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099135219999999993, 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:48.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175783+0000", 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179929+0000", 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041461989999999997 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179943+0000", 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4731e-05 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179953+0000", 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180049+0000", 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6909000000000003e-05 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180073+0000", 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4077e-05 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185641+0000", 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055674330000000001 2024-09-13T11:33:48.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185695+0000", 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4273e-05 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:258 4.3 4:c717f8db:::smithi20438004-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.070231953000004, 2024-09-13T11:33:48.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017162947000000001, 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 258 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213551+0000", 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0760000000000001e-06 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213566+0000", 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5441999999999999e-05 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213585+0000", 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8623000000000001e-05 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213600+0000", 2024-09-13T11:33:48.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4783999999999999e-05 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213667+0000", 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213747+0000", 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9794000000000002e-05 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:48.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228488+0000", 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014740964 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229259+0000", 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077144599999999996 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229743+0000", 2024-09-13T11:33:48.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048437000000000003 2024-09-13T11:33:48.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230652+0000", 2024-09-13T11:33:48.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090904600000000005 2024-09-13T11:33:48.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230688+0000", 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5474e-05 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230712+0000", 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3954000000000001e-05 2024-09-13T11:33:48.326 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:259 4.1 4:8533c921:::smithi20438004-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.065956350999997, 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014548484, 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.327 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 259 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:48.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217826+0000", 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6360000000000001e-06 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217833+0000", 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8700000000000003e-06 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217844+0000", 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217851+0000", 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217896+0000", 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4808000000000002e-05 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217954+0000", 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7856000000000003e-05 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229813+0000", 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011859533 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231353+0000", 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001539503 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232188+0000", 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083500600000000001 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232284+0000", 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6645999999999996e-05 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232325+0000", 2024-09-13T11:33:48.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0071999999999999e-05 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232373+0000", 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8498999999999998e-05 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:266 4.4 4:2451796c:::smithi20438004-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.061456555000007, 2024-09-13T11:33:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099464859999999992, 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 266 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222325+0000", 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222330+0000", 2024-09-13T11:33:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999997e-06 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222341+0000", 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0528e-05 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222347+0000", 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6409999999999996e-06 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222406+0000", 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8449000000000002e-05 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222487+0000", 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1167000000000004e-05 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230197+0000", 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077105120000000001 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230937+0000", 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000740159 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231296+0000", 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035862399999999999 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232186+0000", 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089002899999999997 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232234+0000", 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7414999999999999e-05 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232271+0000", 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7323000000000002e-05 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:283 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.05032842, 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011861159, 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 283 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233453+0000", 2024-09-13T11:33:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233457+0000", 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1150000000000004e-06 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233465+0000", 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233471+0000", 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3450000000000002e-06 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233526+0000", 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4790000000000002e-05 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233602+0000", 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6253000000000003e-05 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237634+0000", 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040318580000000001 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238163+0000", 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052849899999999998 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242711+0000", 2024-09-13T11:33:48.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045478530000000001 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245235+0000", 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025239160000000002 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245286+0000", 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1739e-05 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245314+0000", 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7280999999999999e-05 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:284 4.1 4:8b052c81:::smithi20438004-35:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.049978408000001, 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011653294, 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.346 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 284 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233803+0000", 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4600000000000003e-07 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233808+0000", 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:33:48.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233817+0000", 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9870000000000005e-06 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233824+0000", 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8290000000000002e-06 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233874+0000", 2024-09-13T11:33:48.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9877999999999999e-05 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233943+0000", 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238024+0000", 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040810350000000002 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238440+0000", 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041507799999999999 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242785+0000", 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043457890000000001 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245363+0000", 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025773359999999999 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245416+0000", 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3156999999999999e-05 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245456+0000", 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0000000000000003e-05 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.352 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:286 4.5 4:a9b0c9bd:::smithi20438004-37:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.048996646999996, 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013081485, 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 286 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234785+0000", 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234790+0000", 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.651e-06 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234800+0000", 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234807+0000", 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.861e-06 2024-09-13T11:33:48.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234868+0000", 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0582999999999999e-05 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234967+0000", 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:48.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054743730000000003 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241205+0000", 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076331700000000001 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241351+0000", 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619399999999999 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247766+0000", 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064151290000000003 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247825+0000", 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8739999999999997e-05 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247866+0000", 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747000000000001e-05 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.358 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:290 4.1 4:8d6fdfdb:::smithi20438004-41:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.044442497999995, 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014414148999999999, 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.359 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 290 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239339+0000", 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239344+0000", 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2290000000000002e-06 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239354+0000", 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239361+0000", 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.092e-06 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239406+0000", 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126000000000001e-05 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239452+0000", 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6192000000000001e-05 2024-09-13T11:33:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248478+0000", 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090262209999999992 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249309+0000", 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083057300000000003 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253400+0000", 2024-09-13T11:33:48.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040908760000000002 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253662+0000", 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026229500000000001 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253715+0000", 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2836e-05 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253753+0000", 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7526e-05 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:292 4.1 4:84fa1f64:::smithi20438004-43:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.044019285000005, 2024-09-13T11:33:48.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010951834000000001, 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 292 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8800000000000002e-07 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239765+0000", 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239773+0000", 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8229999999999995e-06 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239779+0000", 2024-09-13T11:33:48.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9900000000000002e-06 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239831+0000", 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2326e-05 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239910+0000", 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8623999999999998e-05 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245574+0000", 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005663927 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.246099+0000", 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052478300000000002 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247963+0000", 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001864166 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250625+0000", 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002662228 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250676+0000", 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0989000000000003e-05 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250713+0000", 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7360000000000001e-05 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.371 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:317 4.1 4:8c30e60e:::smithi20438004-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.032895379999999, 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013134860999999999, 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.372 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 317 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250892+0000", 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250900+0000", 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:33:48.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250908+0000", 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.337e-06 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250967+0000", 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8201000000000001e-05 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251045+0000", 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299000000000004e-05 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258769+0000", 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007724323 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259599+0000", 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082958200000000004 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260992+0000", 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00139291 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263918+0000", 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029263190000000001 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263971+0000", 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2670999999999998e-05 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264020+0000", 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:330 4.3 4:c7f26bfe:::smithi20438004-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.378 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.031672564999994, 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011943621, 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 330 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252109+0000", 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252116+0000", 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9199999999999998e-06 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252129+0000", 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2768e-05 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252138+0000", 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5939999999999994e-06 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252205+0000", 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252280+0000", 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4930000000000003e-05 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259484+0000", 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00720434 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259994+0000", 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051054200000000001 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260333+0000", 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033855000000000002 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263947+0000", 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00361452 2024-09-13T11:33:48.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264015+0000", 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7155000000000003e-05 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264052+0000", 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305000000000003e-05 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:333 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.031382368999999, 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011814712, 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 333 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.385 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:48.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252406+0000", 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252416+0000", 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-05 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252423+0000", 2024-09-13T11:33:48.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252486+0000", 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2805999999999995e-05 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252540+0000", 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3897000000000003e-05 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259138+0000", 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065977429999999997 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259785+0000", 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064727700000000003 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264056+0000", 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042706710000000002 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264152+0000", 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5877000000000004e-05 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264180+0000", 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8674999999999999e-05 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264213+0000", 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3015000000000002e-05 2024-09-13T11:33:48.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:433 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.006831521999999, 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098551839999999995, 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276950+0000", 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999995e-07 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276955+0000", 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1059999999999999e-06 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276965+0000", 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276977+0000", 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1518000000000001e-05 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277035+0000", 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8133000000000001e-05 2024-09-13T11:33:48.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277125+0000", 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0148999999999994e-05 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282235+0000", 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005109461 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282725+0000", 2024-09-13T11:33:48.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049045700000000002 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286634+0000", 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039087929999999998 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286707+0000", 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2692000000000001e-05 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286772+0000", 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5828999999999996e-05 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286805+0000", 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2134000000000001e-05 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:791 4.5 4:a5f8f9e2:::smithi20438004-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:48.397 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 80.583948612, 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019917450999999999, 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.398 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 791 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699834+0000", 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699840+0000", 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5989999999999998e-06 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699852+0000", 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:33:48.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699861+0000", 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5259999999999993e-06 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699949+0000", 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8083000000000004e-05 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699953+0000", 2024-09-13T11:33:48.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3869999999999997e-06 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701189+0000", 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012354740000000001 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701250+0000", 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1733e-05 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.719750+0000", 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018499504 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1374 4.1 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 72.203881483999993, 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011338608999999999, 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:48.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1374 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079900+0000", 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-07 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079901+0000", 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079905+0000", 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2039999999999999e-06 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079908+0000", 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8200000000000001e-06 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081697+0000", 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017893080000000001 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081802+0000", 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105231 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.091238+0000", 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094355819999999997 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.407 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1454 4.1 4:84fa1f64:::smithi20438004-43:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 71.447834641, 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025598754000000001, 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1454 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835947+0000", 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8500000000000002e-07 2024-09-13T11:33:48.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835948+0000", 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835952+0000", 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4259999999999999e-06 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835954+0000", 2024-09-13T11:33:48.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8550000000000002e-06 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836019+0000", 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4416000000000001e-05 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836271+0000", 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025171299999999998 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.861545+0000", 2024-09-13T11:33:48.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025274496 2024-09-13T11:33:48.412 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.412 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.412 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.412 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.412 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.412 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:48.413 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:48.414 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:48.412+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:48.415 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:48.415 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:48.415 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:48.514 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:48.513+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:48.520 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:48.520 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:48.521 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:48.521 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:48.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:48.615 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:48.614+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:48.631 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:48.632 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:48.632 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:48.640 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:48.715 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:48.714+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 102.39994885900001, 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079674329999999995, 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4344", 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343169+0000", 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343177+0000", 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4819999999999994e-06 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343189+0000", 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2306999999999999e-05 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343199+0000", 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343279+0000", 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0025999999999996e-05 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343289+0000", 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0518e-05 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348428+0000", 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051380779999999999 2024-09-13T11:33:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348510+0000", 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2226999999999997e-05 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348598+0000", 2024-09-13T11:33:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8722999999999997e-05 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349014+0000", 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041537900000000001 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349279+0000", 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026517700000000001 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351081+0000", 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001802292 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351110+0000", 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8651000000000001e-05 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351132+0000", 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069000000000002e-05 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:21 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'4, mlcod=38'2)", 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.229965806999999, 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002707297, 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513149+0000", 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.322e-06 2024-09-13T11:33:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514452+0000", 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303153 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514468+0000", 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6045999999999998e-05 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514478+0000", 2024-09-13T11:33:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4960000000000002e-06 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514531+0000", 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3442000000000001e-05 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514546+0000", 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5211999999999999e-05 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515821+0000", 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001274854 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515855+0000", 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3772000000000001e-05 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.721739309, 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002340637, 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021376+0000", 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.013e-06 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022454+0000", 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010781110000000001 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022469+0000", 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4199000000000001e-05 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022477+0000", 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022532+0000", 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5636000000000001e-05 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022548+0000", 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5769000000000001e-05 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023686+0000", 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011379929999999999 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023715+0000", 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8659000000000001e-05 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:182 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'8, mlcod=38'6)", 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.715673636000005, 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022898689999999999, 2024-09-13T11:33:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027441+0000", 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7599999999999996e-07 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028556+0000", 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011154439999999999 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028566+0000", 2024-09-13T11:33:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8269999999999993e-06 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028571+0000", 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8550000000000001e-06 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028614+0000", 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028629+0000", 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029703+0000", 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001073898 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029730+0000", 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6993999999999999e-05 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:190 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'11, mlcod=38'9)", 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.685845907000001, 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022963139999999998, 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058460+0000", 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192238 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058470+0000", 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058475+0000", 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5399999999999997e-06 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058514+0000", 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8887999999999998e-05 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058526+0000", 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1683e-05 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059537+0000", 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001011907 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059564+0000", 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.348662599999997, 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048595679999999999, 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394452+0000", 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397390+0000", 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002937977 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397401+0000", 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1172000000000001e-05 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397407+0000", 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397452+0000", 2024-09-13T11:33:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4705e-05 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397467+0000", 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5622000000000001e-05 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399273+0000", 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001805823 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399310+0000", 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7512999999999998e-05 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1363 4.2 e43/31 4:4ba9f189:::smithi20438004-14:head v 43'50, mlcod=43'48)", 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 72.697530166999996, 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021379849999999998, 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045585+0000", 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.604e-06 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045958+0000", 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000372672 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045978+0000", 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0262999999999999e-05 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045991+0000", 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2811e-05 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046079+0000", 2024-09-13T11:33:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8667999999999999e-05 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046103+0000", 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4131e-05 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047664+0000", 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001560098 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047721+0000", 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7738000000000002e-05 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1410 4.4 e43/41 4:2451796c:::smithi20438004-17:head v 43'44, mlcod=43'42)", 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 72.614557658999999, 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022888489999999999, 2024-09-13T11:33:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128557+0000", 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3939999999999999e-06 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129014+0000", 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045637100000000001 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129032+0000", 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8685000000000002e-05 2024-09-13T11:33:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129042+0000", 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129100+0000", 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8238e-05 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129119+0000", 2024-09-13T11:33:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8258e-05 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130809+0000", 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001690045 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130845+0000", 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5991000000000002e-05 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 71.711967752999996, 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071247150000000002, 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031147+0000", 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.579e-06 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036431+0000", 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00528394 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036449+0000", 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7329e-05 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036457+0000", 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036529+0000", 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1506000000000003e-05 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036551+0000", 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2154000000000002e-05 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038234+0000", 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001683192 2024-09-13T11:33:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038270+0000", 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6408000000000001e-05 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1679 4.4 4.9c0290d4 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 66.683910049000005, 2024-09-13T11:33:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47635528100000002, 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1679 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059205+0000", 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.812e-06 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059211+0000", 2024-09-13T11:33:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1469999999999998e-06 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059222+0000", 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0560000000000001e-05 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059233+0000", 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0591000000000001e-05 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059286+0000", 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519e-05 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059293+0000", 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4690000000000002e-06 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535521+0000", 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47622830100000002 2024-09-13T11:33:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535559+0000", 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7882000000000001e-05 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.340662418999997, 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002436952, 2024-09-13T11:33:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402453+0000", 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.632e-06 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403268+0000", 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081553400000000003 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403289+0000", 2024-09-13T11:33:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0353e-05 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403298+0000", 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4760000000000004e-06 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403369+0000", 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0858000000000005e-05 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403397+0000", 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8518e-05 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404841+0000", 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001443228 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404888+0000", 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7352999999999999e-05 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.335136022999997, 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003287178, 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407979+0000", 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409585+0000", 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016060479999999999 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409598+0000", 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3027000000000001e-05 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409606+0000", 2024-09-13T11:33:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2190000000000004e-06 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409676+0000", 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9791999999999999e-05 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409707+0000", 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1692000000000001e-05 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411223+0000", 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001515705 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411265+0000", 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1649000000000001e-05 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1705 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'62, mlcod=0'0)", 2024-09-13T11:33:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.327531622999999, 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003138161, 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415583+0000", 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9699999999999995e-07 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416900+0000", 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001317251 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416926+0000", 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5622e-05 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416942+0000", 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6741000000000001e-05 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417014+0000", 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1298999999999996e-05 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417042+0000", 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.851e-05 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418673+0000", 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016310949999999999 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418720+0000", 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6845999999999998e-05 2024-09-13T11:33:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1706 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'63, mlcod=45'61)", 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.322199425999997, 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003090597, 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420915+0000", 2024-09-13T11:33:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-06 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422306+0000", 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001390753 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422317+0000", 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422322+0000", 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4360000000000001e-06 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422379+0000", 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7003000000000003e-05 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422405+0000", 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6424e-05 2024-09-13T11:33:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.423958+0000", 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015526680000000001 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.424005+0000", 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6619000000000002e-05 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1919 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'63, mlcod=53'61)", 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 58.776999447999998, 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002758908, 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966116+0000", 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0370000000000001e-06 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967135+0000", 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010185260000000001 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967149+0000", 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3903999999999999e-05 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967157+0000", 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.405e-06 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967255+0000", 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8441000000000002e-05 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967280+0000", 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5079000000000002e-05 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968828+0000", 2024-09-13T11:33:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001547936 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968873+0000", 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4579999999999997e-05 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 58.769048724999998, 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028733999999999999, 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974066+0000", 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975332+0000", 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012659349999999999 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975345+0000", 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3402e-05 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975351+0000", 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4450000000000001e-06 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975438+0000", 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6687999999999994e-05 2024-09-13T11:33:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975457+0000", 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8913e-05 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976901+0000", 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014437180000000001 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976938+0000", 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7441999999999998e-05 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.823914960000003, 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022722770000000001, 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919200+0000", 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9000000000000005e-07 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919903+0000", 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070307200000000001 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919915+0000", 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2482e-05 2024-09-13T11:33:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919922+0000", 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5459999999999997e-06 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919988+0000", 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6377999999999994e-05 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.920027+0000", 2024-09-13T11:33:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8649999999999998e-05 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921419+0000", 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013924530000000001 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921471+0000", 2024-09-13T11:33:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1706000000000003e-05 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2077 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'67, mlcod=53'65)", 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.815772328999998, 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026288430000000001, 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927342+0000", 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928463+0000", 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011212640000000001 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928476+0000", 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3147e-05 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928484+0000", 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928549+0000", 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5001999999999999e-05 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928565+0000", 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6124999999999999e-05 2024-09-13T11:33:48.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929914+0000", 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001348859 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929970+0000", 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5912000000000003e-05 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:3290 4.4 e74/56 4:2451796c:::smithi20438004-17:head v 74'77, mlcod=73'75)", 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.136767989999999, 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025948210000000002, 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606347+0000", 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607289+0000", 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094240700000000003 2024-09-13T11:33:48.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607322+0000", 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3040999999999998e-05 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607350+0000", 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7331999999999999e-05 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607464+0000", 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114826 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607491+0000", 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6271e-05 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.608888+0000", 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013977309999999999 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.608940+0000", 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1826000000000002e-05 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:3291 4.4 e74/56 4:2451796c:::smithi20438004-17:head v 74'78, mlcod=73'76)", 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.131650692000001, 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022192150000000001, 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:48.838 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611464+0000", 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612224+0000", 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076036799999999998 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612238+0000", 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.342e-05 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612244+0000", 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4949999999999996e-06 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612295+0000", 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0392000000000001e-05 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612314+0000", 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9568e-05 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.613631+0000", 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013171019999999999 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.613682+0000", 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0841000000000003e-05 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:48.842 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:48.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:48.845 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:48.843+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:48.945 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:48.944+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:48.947 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:48.947 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:48.947 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:48.947 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:48.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:49.046 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:49.044+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:49.097 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:49.097 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:49.097 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:49.098 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:49.098 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:49.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:49.150 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:49.149+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:49.211 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:49.211 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:49.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:49.211 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:49.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:206 3.1 e38/30 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.117114315999999, 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012415890000000001, 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:49.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092191+0000", 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.092192+0000", 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.63e-07 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100358+0000", 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081664059999999993 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100384+0000", 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5287e-05 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100401+0000", 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7744000000000002e-05 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100478+0000", 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6769999999999999e-05 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100508+0000", 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0134e-05 2024-09-13T11:33:49.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104514+0000", 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040060590000000002 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104607+0000", 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3027000000000002e-05 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:201 3.1 e38/30 3:b265d015:::smithi20438004-2 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.108868977, 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012387119, 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.217 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100437+0000", 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100438+0000", 2024-09-13T11:33:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1000000000000001e-06 2024-09-13T11:33:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108964+0000", 2024-09-13T11:33:49.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085255959999999999 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108984+0000", 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0466e-05 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108996+0000", 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2149e-05 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109097+0000", 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010042799999999999 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109121+0000", 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.455e-05 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112773+0000", 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036519780000000002 2024-09-13T11:33:49.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112824+0000", 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0852e-05 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.100247506000002, 2024-09-13T11:33:49.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096300529999999995, 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109058+0000", 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109060+0000", 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2279999999999999e-06 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116086+0000", 2024-09-13T11:33:49.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070260239999999996 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116100+0000", 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4209999999999999e-05 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116109+0000", 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6569999999999992e-06 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116160+0000", 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0470000000000003e-05 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116191+0000", 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1494e-05 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118639+0000", 2024-09-13T11:33:49.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024481030000000001 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.118688+0000", 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8866999999999998e-05 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.228 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:49.228 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:49.228 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.093175711000001, 2024-09-13T11:33:49.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010922463, 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116130+0000", 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116131+0000", 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6899999999999998e-07 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124407+0000", 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082764919999999999 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124422+0000", 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5404999999999999e-05 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124432+0000", 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8910000000000007e-06 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124485+0000", 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2750999999999997e-05 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.124508+0000", 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3076999999999999e-05 2024-09-13T11:33:49.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127004+0000", 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024962930000000001 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.127053+0000", 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8084999999999998e-05 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.079511612999994, 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091903760000000001, 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.234 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129794+0000", 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.129795+0000", 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8700000000000001e-07 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135325+0000", 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055302190000000003 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135340+0000", 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4977000000000001e-05 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135351+0000", 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1358999999999999e-05 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135418+0000", 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000002e-05 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135447+0000", 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8415000000000002e-05 2024-09-13T11:33:49.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138936+0000", 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034896800000000002 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.138985+0000", 2024-09-13T11:33:49.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8338999999999999e-05 2024-09-13T11:33:49.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.239 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.240 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:49.240 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:49.240 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.069316541000006, 2024-09-13T11:33:49.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010981263, 2024-09-13T11:33:49.240 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139989+0000", 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.139990+0000", 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2e-07 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147357+0000", 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073669950000000003 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147376+0000", 2024-09-13T11:33:49.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9453000000000002e-05 2024-09-13T11:33:49.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147388+0000", 2024-09-13T11:33:49.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1527e-05 2024-09-13T11:33:49.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147472+0000", 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4630999999999997e-05 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147493+0000", 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0687e-05 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150899+0000", 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034054739999999999 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.150971+0000", 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1976000000000003e-05 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:49.246 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.061865533000002, 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015742025999999999, 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147440+0000", 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147441+0000", 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7700000000000004e-07 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157019+0000", 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095782090000000007 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157057+0000", 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157072+0000", 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5082999999999999e-05 2024-09-13T11:33:49.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157157+0000", 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5065e-05 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157188+0000", 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0920000000000002e-05 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163133+0000", 2024-09-13T11:33:49.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059446890000000004 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.163182+0000", 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9564999999999997e-05 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.051793445000001, 2024-09-13T11:33:49.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015556076, 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157512+0000", 2024-09-13T11:33:49.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157514+0000", 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167722+0000", 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010208016 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167740+0000", 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7668000000000001e-05 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167754+0000", 2024-09-13T11:33:49.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4647999999999999e-05 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167831+0000", 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7229000000000002e-05 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167855+0000", 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3467000000000001e-05 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173001+0000", 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051458160000000001 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.173068+0000", 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7720999999999998e-05 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:49.256 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.041524953000007, 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011370243, 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167781+0000", 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3499999999999996e-07 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175562+0000", 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077811060000000003 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175578+0000", 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5486e-05 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175587+0000", 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9509999999999995e-06 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175668+0000", 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1224999999999996e-05 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175695+0000", 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6505000000000001e-05 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179075+0000", 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003380508 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179151+0000", 2024-09-13T11:33:49.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5927000000000007e-05 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:49.262 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.033698291999997, 2024-09-13T11:33:49.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017514140000000001, 2024-09-13T11:33:49.263 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.263 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.263 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.263 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.263 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175608+0000", 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188284+0000", 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012675702 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188299+0000", 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5122e-05 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188308+0000", 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9819999999999997e-06 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188367+0000", 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8774999999999999e-05 2024-09-13T11:33:49.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188387+0000", 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0649000000000002e-05 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193056+0000", 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046684830000000002 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193122+0000", 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5790999999999994e-05 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.269 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.020973314000003, 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015855429000000001, 2024-09-13T11:33:49.270 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.271 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.271 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.271 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188332+0000", 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188333+0000", 2024-09-13T11:33:49.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-07 2024-09-13T11:33:49.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199600+0000", 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011266717000000001 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199613+0000", 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3682e-05 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199620+0000", 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1339999999999998e-06 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199676+0000", 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5689000000000002e-05 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199708+0000", 2024-09-13T11:33:49.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2082999999999998e-05 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204123+0000", 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004414734 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.204188+0000", 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.009664776999998, 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092010560000000009, 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:49.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199641+0000", 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.199642+0000", 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9599999999999999e-07 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.278 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205539+0000", 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058972290000000004 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205553+0000", 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4577e-05 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205564+0000", 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0254e-05 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205630+0000", 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6057999999999997e-05 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205658+0000", 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7979000000000001e-05 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208792+0000", 2024-09-13T11:33:49.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031342499999999999 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208842+0000", 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0213000000000002e-05 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:245 3.1 e38/30 3:adf25eb4:::smithi20438004-46 low_tier:head v 38'15, mlcod=38'11)", 2024-09-13T11:33:49.281 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.003718448000001, 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011146911000000001, 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205587+0000", 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.205588+0000", 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3399999999999997e-07 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213344+0000", 2024-09-13T11:33:49.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077554440000000002 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213359+0000", 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5455999999999999e-05 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213369+0000", 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9890000000000004e-06 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213453+0000", 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4153999999999994e-05 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213479+0000", 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5831999999999999e-05 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216694+0000", 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032149990000000001 2024-09-13T11:33:49.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.216734+0000", 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0102999999999999e-05 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:260 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.989947427000004, 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011239344, 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 260 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219358+0000", 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219360+0000", 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2449999999999999e-06 2024-09-13T11:33:49.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219366+0000", 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7179999999999999e-06 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219376+0000", 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9990000000000003e-06 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219382+0000", 2024-09-13T11:33:49.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8150000000000002e-06 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219428+0000", 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5751999999999998e-05 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.219492+0000", 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4119000000000001e-05 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:49.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227887+0000", 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083953989999999996 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228998+0000", 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001110764 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229290+0000", 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029145400000000002 2024-09-13T11:33:49.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230489+0000", 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001199786 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230548+0000", 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8900999999999997e-05 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230598+0000", 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9391999999999997e-05 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:249 3.1 e38/30 3:81faf980:::smithi20438004-50 low_tier:head v 38'21, mlcod=38'18)", 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.986671290999993, 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091134460000000007, 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.293 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222635+0000", 2024-09-13T11:33:49.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.68e-07 2024-09-13T11:33:49.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227843+0000", 2024-09-13T11:33:49.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005207494 2024-09-13T11:33:49.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227853+0000", 2024-09-13T11:33:49.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117e-05 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227860+0000", 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4880000000000001e-06 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227912+0000", 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1900000000000001e-05 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.227933+0000", 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0474000000000001e-05 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231709+0000", 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037762759999999999 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231748+0000", 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9029000000000002e-05 2024-09-13T11:33:49.298 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:278 4.2 4:4d4c8512:::smithi20438004-29:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.978311691000002, 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091148359999999994, 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.299 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230994+0000", 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230995+0000", 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4200000000000004e-07 2024-09-13T11:33:49.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231000+0000", 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3290000000000001e-06 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231017+0000", 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6242000000000001e-05 2024-09-13T11:33:49.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231025+0000", 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9920000000000007e-06 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231094+0000", 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9554999999999994e-05 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231171+0000", 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7082999999999994e-05 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.236799+0000", 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056282260000000001 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237539+0000", 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073930700000000003 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237574+0000", 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5191999999999998e-05 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240013+0000", 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002439185 2024-09-13T11:33:49.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240070+0000", 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6650999999999997e-05 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240109+0000", 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9131999999999997e-05 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:287 4.2 4:4297cb20:::smithi20438004-38:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.974376219000007, 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013317569, 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 287 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.307 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234930+0000", 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234931+0000", 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.192e-06 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234936+0000", 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0640000000000001e-06 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234953+0000", 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7292e-05 2024-09-13T11:33:49.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234961+0000", 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0590000000000003e-06 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235026+0000", 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5073000000000005e-05 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.235111+0000", 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4635999999999994e-05 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242149+0000", 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070384990000000001 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242840+0000", 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069074700000000002 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242936+0000", 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5730000000000007e-05 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248138+0000", 2024-09-13T11:33:49.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052018330000000003 2024-09-13T11:33:49.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248197+0000", 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9512000000000002e-05 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248247+0000", 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.315 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:310 4.2 4:42f22b3e:::smithi20438004-11:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.315 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:49.315 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.960449851000007, 2024-09-13T11:33:49.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095515259999999994, 2024-09-13T11:33:49.315 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.316 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.316 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.316 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.316 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.316 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 310 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248856+0000", 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248857+0000", 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-06 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248863+0000", 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6529999999999999e-06 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248874+0000", 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.093e-05 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248887+0000", 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3009e-05 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248932+0000", 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5445999999999998e-05 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249025+0000", 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3098999999999998e-05 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251503+0000", 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024772629999999999 2024-09-13T11:33:49.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252148+0000", 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064507900000000001 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252433+0000", 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028559100000000002 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258304+0000", 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058703100000000001 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258361+0000", 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7658999999999998e-05 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258407+0000", 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6319999999999997e-05 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.322 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:313 4.2 4:4ba9f189:::smithi20438004-14:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 81.959571143999995, 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013958247999999999, 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.323 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 313 2024-09-13T11:33:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249735+0000", 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249736+0000", 2024-09-13T11:33:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9999999999999996e-07 2024-09-13T11:33:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249741+0000", 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5929999999999996e-06 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249750+0000", 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7369999999999995e-06 2024-09-13T11:33:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249757+0000", 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5360000000000004e-06 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249806+0000", 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9018999999999999e-05 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249882+0000", 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5943000000000001e-05 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,5", 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258629+0000", 2024-09-13T11:33:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087464589999999998 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259299+0000", 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067041799999999999 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259523+0000", 2024-09-13T11:33:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000223964 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263636+0000", 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041130539999999997 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263669+0000", 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2849999999999999e-05 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263693+0000", 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3774999999999999e-05 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1437 4.2 4:4b088bf8:::smithi20438004-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 72.405330758000005, 2024-09-13T11:33:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014642921, 2024-09-13T11:33:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1437 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803975+0000", 2024-09-13T11:33:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803976+0000", 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.35e-06 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803984+0000", 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2080000000000003e-06 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.803995+0000", 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1148999999999999e-05 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804001+0000", 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2870000000000001e-06 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804096+0000", 2024-09-13T11:33:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5424e-05 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.804165+0000", 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8596000000000005e-05 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.818618+0000", 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452906999999999 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.338 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:49.340 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:49.341 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:49.338+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:49.440 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:49.439+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:49.484 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:49.485 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:49.485 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:49.485 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:49.486 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:49.486 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:49.486 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:49.487 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-13T11:33:49.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-13T11:33:49.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:49.498 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:33:49.532 INFO:tasks.rados.rados.0.smithi204.stdout:1851: set_chunk oid 49 offset: 39936 length: 1781760 target oid 64b0ec958f46e0bde068907626863799a07e7e5b874bd667875147d240d0f7ef offset: 0 snap -1 2024-09-13T11:33:49.532 INFO:tasks.rados.rados.0.smithi204.stdout:1852: write oid 30 current snap is 0 2024-09-13T11:33:49.532 INFO:tasks.rados.rados.0.smithi204.stdout:1850: finishing write tid 1 to smithi20438004-26 2024-09-13T11:33:49.532 INFO:tasks.rados.rados.0.smithi204.stdout:1852: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-13T11:33:49.537 INFO:tasks.rados.rados.0.smithi204.stdout:1852: writing smithi20438004-30 from 705993 to 1180700 tid 1 2024-09-13T11:33:49.541 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:49.540+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:49.544 INFO:tasks.rados.rados.0.smithi204.stdout:1852: writing smithi20438004-30 from 1582080 to 2194221 tid 2 2024-09-13T11:33:49.544 INFO:tasks.rados.rados.0.smithi204.stdout:1853: read oid 2 snap -1 2024-09-13T11:33:49.544 INFO:tasks.rados.rados.0.smithi204.stdout:1853: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-13T11:33:49.544 INFO:tasks.rados.rados.0.smithi204.stdout:1850: finishing write tid 2 to smithi20438004-26 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1854: read oid 12 snap -1 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1854: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1855: write oid 5 current snap is 0 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1850: finishing write tid 3 to smithi20438004-26 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1850: finishing write tid 4 to smithi20438004-26 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1850: oid 26 updating version 0 to 78 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1850: oid 26 updating version 78 to 79 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1850: oid 26 updating version 79 to 80 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 80 (ObjNum 398 snap 0 seq_num 398) dirty exists 2024-09-13T11:33:49.545 INFO:tasks.rados.rados.0.smithi204.stdout:1850: left oid 26 (ObjNum 398 snap 0 seq_num 398) 2024-09-13T11:33:49.546 INFO:tasks.rados.rados.0.smithi204.stdout:1855: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-13T11:33:49.551 INFO:tasks.rados.rados.0.smithi204.stdout:1855: writing smithi20438004-5 from 722800 to 1272756 tid 1 2024-09-13T11:33:49.559 INFO:tasks.rados.rados.0.smithi204.stdout:1855: writing smithi20438004-5 from 2040562 to 2722800 tid 2 2024-09-13T11:33:49.560 INFO:tasks.rados.rados.0.smithi204.stdout:1850: done (5 left) 2024-09-13T11:33:49.560 INFO:tasks.rados.rados.0.smithi204.stdout:1856: write oid 6 current snap is 0 2024-09-13T11:33:49.560 INFO:tasks.rados.rados.0.smithi204.stdout:1852: finishing write tid 1 to smithi20438004-30 2024-09-13T11:33:49.560 INFO:tasks.rados.rados.0.smithi204.stdout:1856: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-13T11:33:49.567 INFO:tasks.rados.rados.0.smithi204.stdout:1856: writing smithi20438004-6 from 739607 to 1364812 tid 1 2024-09-13T11:33:49.573 INFO:tasks.rados.rados.0.smithi204.stdout:1856: writing smithi20438004-6 from 1982691 to 2482148 tid 2 2024-09-13T11:33:49.574 INFO:tasks.rados.rados.0.smithi204.stdout:1857: read oid 1 snap -1 2024-09-13T11:33:49.574 INFO:tasks.rados.rados.0.smithi204.stdout:1857: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-13T11:33:49.574 INFO:tasks.rados.rados.0.smithi204.stdout:1858: write oid 39 current snap is 0 2024-09-13T11:33:49.574 INFO:tasks.rados.rados.0.smithi204.stdout:1858: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-13T11:33:49.581 INFO:tasks.rados.rados.0.smithi204.stdout:1858: writing smithi20438004-39 from 756414 to 1456868 tid 1 2024-09-13T11:33:49.589 INFO:tasks.rados.rados.0.smithi204.stdout:1858: writing smithi20438004-39 from 2041173 to 2756414 tid 2 2024-09-13T11:33:49.589 INFO:tasks.rados.rados.0.smithi204.stdout:1859: read oid 20 snap -1 2024-09-13T11:33:49.590 INFO:tasks.rados.rados.0.smithi204.stdout:1859: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-13T11:33:49.590 INFO:tasks.rados.rados.0.smithi204.stdout:1860: read oid 14 snap -1 2024-09-13T11:33:49.590 INFO:tasks.rados.rados.0.smithi204.stdout:1860: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-13T11:33:49.590 INFO:tasks.rados.rados.0.smithi204.stdout:1861: read oid 24 snap -1 2024-09-13T11:33:49.590 INFO:tasks.rados.rados.0.smithi204.stdout:1861: expect (ObjNum 363 snap 0 seq_num 363) 2024-09-13T11:33:49.590 INFO:tasks.rados.rados.0.smithi204.stdout:1862: tier_evict oid 40 2024-09-13T11:33:49.591 INFO:tasks.rados.rados.0.smithi204.stdout:1862: tier_evict oid 40 snap -1 2024-09-13T11:33:49.591 INFO:tasks.rados.rados.0.smithi204.stdout:1853: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-13T11:33:49.637 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:49.638 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:49.642 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:49.640+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:49.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:49.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-13T11:33:49.680 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:49.731 INFO:tasks.rados.rados.0.smithi204.stdout:1852: finishing write tid 2 to smithi20438004-30 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1852: finishing write tid 3 to smithi20438004-30 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1852: finishing write tid 4 to smithi20438004-30 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1852: oid 30 updating version 0 to 94 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1852: oid 30 version 94 is already newer than 93 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1852: oid 30 updating version 94 to 95 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 95 (ObjNum 399 snap 0 seq_num 399) dirty exists 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1852: left oid 30 (ObjNum 399 snap 0 seq_num 399) 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1851: finishing set_chunk to oid 49 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 44 (ObjNum 382 snap 0 seq_num 382) dirty exists 2024-09-13T11:33:49.732 INFO:tasks.rados.rados.0.smithi204.stdout:1854: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-13T11:33:49.743 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:49.742+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4263.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e24)", 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 105.375271395, 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27446322899999998, 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.750 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4263", 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/2959334534", 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372238+0000", 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1100000000000005e-07 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372244+0000", 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.66e-06 2024-09-13T11:33:49.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372256+0000", 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1817000000000001e-05 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372264+0000", 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0220000000000004e-06 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372320+0000", 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6437999999999997e-05 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.372329+0000", 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2789999999999998e-06 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644121+0000", 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27179224899999999 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644271+0000", 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014990000000000001 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.644421+0000", 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015035600000000001 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646446+0000", 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00202454 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646524+0000", 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8839999999999997e-05 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646598+0000", 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3862999999999994e-05 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646654+0000", 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5609999999999998e-05 2024-09-13T11:33:49.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:04.646701+0000", 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6844000000000001e-05 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:207 3.2 3:59a396eb:::smithi20438004-8 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.662874348000003, 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063519737000000007, 2024-09-13T11:33:49.757 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 207 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084635+0000", 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7099999999999998e-07 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084636+0000", 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.559e-06 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084640+0000", 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.118e-06 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084644+0000", 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8640000000000004e-06 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084687+0000", 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2295000000000001e-05 2024-09-13T11:33:49.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084780+0000", 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2880000000000005e-05 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.106902+0000", 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022122822 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.112195+0000", 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052927 2024-09-13T11:33:49.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.121161+0000", 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089661740000000004 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148052+0000", 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026890595 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148105+0000", 2024-09-13T11:33:49.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2673000000000002e-05 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148154+0000", 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9786000000000001e-05 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:208 3.2 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.662855632000003, 2024-09-13T11:33:49.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.049409478, 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 208 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084653+0000", 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084654+0000", 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3500000000000002e-07 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084657+0000", 2024-09-13T11:33:49.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1769999999999998e-06 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084663+0000", 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8329999999999999e-06 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084668+0000", 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6200000000000004e-06 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084884+0000", 2024-09-13T11:33:49.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021599499999999999 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.085046+0000", 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016192600000000001 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.094613+0000", 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095668200000000002 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.101900+0000", 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0072866160000000001 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.110249+0000", 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008349034 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133903+0000", 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023654272 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.133976+0000", 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2581000000000002e-05 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134063+0000", 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7169000000000005e-05 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:211 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.662814603000001, 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047621726000000003, 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 211 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084694+0000", 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084695+0000", 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6100000000000002e-07 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084697+0000", 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2419999999999999e-06 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8160000000000001e-06 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084706+0000", 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.313e-06 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086622+0000", 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001916212 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086720+0000", 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8077000000000003e-05 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.100598+0000", 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013877195 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.104958+0000", 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043607899999999998 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115608+0000", 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010650138 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132197+0000", 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016589008999999998 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132257+0000", 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9948000000000003e-05 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.132316+0000", 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8625000000000002e-05 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:212 3.3 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.662797022999996, 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.063179242999999996, 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 212 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084712+0000", 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7300000000000002e-07 2024-09-13T11:33:49.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.255e-06 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084719+0000", 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4800000000000003e-06 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.084724+0000", 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.527e-06 2024-09-13T11:33:49.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086839+0000", 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021155240000000001 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.086947+0000", 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107675 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.109128+0000", 2024-09-13T11:33:49.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022180947999999999 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.113055+0000", 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039269029999999998 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.119148+0000", 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060933810000000001 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147786+0000", 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028638395000000001 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147838+0000", 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1405e-05 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.147891+0000", 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3377000000000001e-05 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:216 3.2 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.656857775999995, 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055588580999999998, 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 216 2024-09-13T11:33:49.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090651+0000", 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090653+0000", 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090663+0000", 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0708e-05 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090675+0000", 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1389000000000001e-05 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.090688+0000", 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3193000000000001e-05 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096478+0000", 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057898150000000002 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.096694+0000", 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021582700000000001 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.103295+0000", 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006601048 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.108559+0000", 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052645449999999998 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.114944+0000", 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063845389999999998 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146121+0000", 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031177449999999999 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146186+0000", 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5235999999999997e-05 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.146240+0000", 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3368000000000001e-05 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:220 3.2 3:5569bbd7:::smithi20438004-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.631682252000004, 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.033838506999999997, 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.788 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 220 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115827+0000", 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115828+0000", 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115832+0000", 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.746e-06 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115842+0000", 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2010000000000005e-06 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115847+0000", 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9740000000000001e-06 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.115897+0000", 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0114999999999997e-05 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.116103+0000", 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020598799999999999 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.135329+0000", 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019226422999999999 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141061+0000", 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057316709999999998 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.142490+0000", 2024-09-13T11:33:49.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001428805 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149552+0000", 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070625610000000002 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149602+0000", 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9826e-05 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149665+0000", 2024-09-13T11:33:49.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3168000000000002e-05 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:226 3.3 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.613130804999997, 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.062274747999999998, 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.794 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 226 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134378+0000", 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134379+0000", 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1850000000000001e-06 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134392+0000", 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2649999999999999e-05 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134404+0000", 2024-09-13T11:33:49.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1615e-05 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134411+0000", 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3509999999999996e-06 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134474+0000", 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2673000000000001e-05 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.134662+0000", 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018810100000000001 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.149047+0000", 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014384902 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.154919+0000", 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058721379999999998 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.171898+0000", 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016979300999999999 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196539+0000", 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024641252999999998 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196614+0000", 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5160999999999994e-05 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.196653+0000", 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8417999999999997e-05 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:228 3.3 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.606848272999997, 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038699092999999997, 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140661+0000", 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140662+0000", 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7269999999999999e-06 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140670+0000", 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8630000000000008e-06 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140696+0000", 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5927000000000001e-05 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140708+0000", 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-05 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140804+0000", 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6137999999999993e-05 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140995+0000", 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000191316 2024-09-13T11:33:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.145999+0000", 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050033129999999997 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152237+0000", 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062383320000000001 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160262+0000", 2024-09-13T11:33:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080251520000000007 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179164+0000", 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018901823000000002 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179296+0000", 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013243400000000001 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179360+0000", 2024-09-13T11:33:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3328000000000001e-05 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:229 3.3 3:d73fa370:::smithi20438004-30 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.606714245999996, 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057728689, 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 229 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140795+0000", 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140796+0000", 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9599999999999998e-07 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140800+0000", 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7729999999999999e-06 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140809+0000", 2024-09-13T11:33:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3669999999999997e-06 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.140814+0000", 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0200000000000002e-06 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141203+0000", 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038959299999999998 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.141375+0000", 2024-09-13T11:33:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017155499999999999 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.156365+0000", 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014989954999999999 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162876+0000", 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065109360000000002 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.187965+0000", 2024-09-13T11:33:49.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025088847000000001 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198445+0000", 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010480171 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198487+0000", 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2635000000000001e-05 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.198523+0000", 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5941e-05 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:238 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.594749270999998, 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.059471032, 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.813 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 238 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152760+0000", 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152761+0000", 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152772+0000", 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1122e-05 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152782+0000", 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5179999999999993e-06 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152790+0000", 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7260000000000002e-06 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.152856+0000", 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6304999999999996e-05 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153180+0000", 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000323617 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167201+0000", 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014021264 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.172985+0000", 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057843349999999998 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.188702+0000", 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015716765000000001 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212106+0000", 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023403714999999999 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212169+0000", 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3224000000000003e-05 2024-09-13T11:33:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.212231+0000", 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1841999999999995e-05 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:240 3.2 3:41bb2de3:::smithi20438004-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.587259294999996, 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034186595, 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 240 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160250+0000", 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160251+0000", 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.562e-06 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160258+0000", 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6640000000000001e-06 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160269+0000", 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.152e-05 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160277+0000", 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4939999999999997e-06 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160327+0000", 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0268000000000001e-05 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.160497+0000", 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017018900000000001 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.164999+0000", 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045015929999999999 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168431+0000", 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003431698 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.184172+0000", 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015741852000000001 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194334+0000", 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010161646999999999 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194390+0000", 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6218999999999998e-05 2024-09-13T11:33:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.194436+0000", 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5889000000000001e-05 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:242 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 18] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.578871488000004, 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053252359999999999, 2024-09-13T11:33:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 242 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168637+0000", 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168640+0000", 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7470000000000001e-06 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168652+0000", 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1909000000000001e-05 2024-09-13T11:33:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168668+0000", 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5888000000000001e-05 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.168680+0000", 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.154e-05 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169643+0000", 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096305100000000003 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000139834 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174850+0000", 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005067841 2024-09-13T11:33:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.178949+0000", 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040991209999999998 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193069+0000", 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014119166000000001 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221801+0000", 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028732905999999999 2024-09-13T11:33:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221852+0000", 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0958000000000002e-05 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221890+0000", 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7398999999999999e-05 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:244 3.2 3:497e8b81:::smithi20438004-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.566931464000007, 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040499786000000003, 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180577+0000", 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180579+0000", 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.499e-06 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180585+0000", 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4649999999999999e-06 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180598+0000", 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2779000000000001e-05 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180604+0000", 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5300000000000004e-06 2024-09-13T11:33:49.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180669+0000", 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4859999999999999e-05 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180840+0000", 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171791 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.190029+0000", 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091888999999999998 2024-09-13T11:33:49.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.193312+0000", 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829579999999999 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.207773+0000", 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014461039 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.220989+0000", 2024-09-13T11:33:49.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013215644 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221038+0000", 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9421e-05 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221077+0000", 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8899999999999997e-05 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:247 3.2 3:4f19c0b8:::smithi20438004-48 low_tier:head [copy-from ver 12] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.561694560000007, 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040924556000000001, 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 247 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185814+0000", 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185816+0000", 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9690000000000001e-06 2024-09-13T11:33:49.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185825+0000", 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6980000000000004e-06 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185842+0000", 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6492000000000001e-05 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185852+0000", 2024-09-13T11:33:49.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0412e-05 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185951+0000", 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9312000000000001e-05 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186223+0000", 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027177000000000001 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.208403+0000", 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022179780999999999 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029604150000000001 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.223543+0000", 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012179840000000001 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226659+0000", 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031159989999999999 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226698+0000", 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9020999999999998e-05 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.226739+0000", 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0846999999999997e-05 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.843 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:248 3.2 3:74f2510c:::smithi20438004-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 82.561371479000002, 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035109013000000001, 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:49.844 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 248 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186137+0000", 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186139+0000", 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.333e-06 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186157+0000", 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8488999999999999e-05 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186170+0000", 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2737e-05 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186181+0000", 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1279e-05 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186356+0000", 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174988 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.186493+0000", 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137045 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,12", 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.203390+0000", 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016897169 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.206673+0000", 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032829510000000001 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.214284+0000", 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076100830000000001 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221141+0000", 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068573740000000003 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221198+0000", 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6916999999999997e-05 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.221246+0000", 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8647999999999999e-05 2024-09-13T11:33:49.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.10.22:30 3.3 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.956334403, 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87173941200000005, 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791175+0000", 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791176+0000", 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4500000000000001e-06 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791183+0000", 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.956e-06 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791193+0000", 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8649999999999996e-06 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791200+0000", 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.148e-06 2024-09-13T11:33:49.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791263+0000", 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3038000000000002e-05 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791370+0000", 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010721999999999999 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.791704+0000", 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033406100000000003 2024-09-13T11:33:49.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.792618+0000", 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091326999999999999 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.793087+0000", 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046985199999999999 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662893+0000", 2024-09-13T11:33:49.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86980523300000001 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662914+0000", 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1319e-05 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.1.22:76 3.3 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.942686890000004, 2024-09-13T11:33:49.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85814986199999999, 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804822+0000", 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804824+0000", 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5889999999999999e-06 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804830+0000", 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7139999999999996e-06 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804845+0000", 2024-09-13T11:33:49.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5158e-05 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804853+0000", 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9349999999999994e-06 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.804910+0000", 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6501e-05 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805017+0000", 2024-09-13T11:33:49.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107237 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.805213+0000", 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019558100000000001 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806246+0000", 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001033225 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.806260+0000", 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4072e-05 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662951+0000", 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85669131700000001 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.662972+0000", 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0533000000000001e-05 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:9 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.902163189000007, 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81776348899999995, 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845346+0000", 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845348+0000", 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4090000000000001e-06 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845360+0000", 2024-09-13T11:33:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1420000000000001e-05 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845389+0000", 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9453000000000001e-05 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845404+0000", 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4919000000000001e-05 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845526+0000", 2024-09-13T11:33:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121588 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845617+0000", 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1119000000000006e-05 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.845789+0000", 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017257000000000001 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846396+0000", 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060634700000000003 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.846771+0000", 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037541099999999997 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663066+0000", 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81629477399999995 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663109+0000", 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3479000000000002e-05 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.3.22:10 3.3 3:de2fec76:::smithi20438004-12 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.898426760000007, 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81407563599999999, 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849082+0000", 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849083+0000", 2024-09-13T11:33:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6300000000000004e-07 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849089+0000", 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849101+0000", 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.167e-05 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849106+0000", 2024-09-13T11:33:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7309999999999998e-06 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849154+0000", 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8077000000000001e-05 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849235+0000", 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0770000000000001e-05 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-09-13T11:33:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.849399+0000", 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000163385 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850247+0000", 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00084889399999999999 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:30.850266+0000", 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8698e-05 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663146+0000", 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81287977300000003 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:31.663158+0000", 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1928e-05 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:49.871 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:49.872 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:49.873 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:49.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:49.874 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-09-13T11:33:49.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-13T11:33:49.875 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:49.872+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:49.940 INFO:tasks.rados.rados.0.smithi204.stdout:1855: finishing write tid 1 to smithi20438004-5 2024-09-13T11:33:49.940 INFO:tasks.rados.rados.0.smithi204.stdout:1855: finishing write tid 2 to smithi20438004-5 2024-09-13T11:33:49.940 INFO:tasks.rados.rados.0.smithi204.stdout:1855: finishing write tid 3 to smithi20438004-5 2024-09-13T11:33:49.940 INFO:tasks.rados.rados.0.smithi204.stdout:1855: finishing write tid 4 to smithi20438004-5 2024-09-13T11:33:49.940 INFO:tasks.rados.rados.0.smithi204.stdout:1855: oid 5 updating version 0 to 91 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1855: oid 5 version 91 is already newer than 89 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1855: oid 5 version 91 is already newer than 90 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 91 (ObjNum 400 snap 0 seq_num 400) dirty exists 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1855: left oid 5 (ObjNum 400 snap 0 seq_num 400) 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1856: finishing write tid 1 to smithi20438004-6 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1851: done (11 left) 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1852: done (10 left) 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1853: done (9 left) 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1854: done (8 left) 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1855: done (7 left) 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1863: tier_promote oid 42 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1856: finishing write tid 2 to smithi20438004-6 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1856: finishing write tid 3 to smithi20438004-6 2024-09-13T11:33:49.941 INFO:tasks.rados.rados.0.smithi204.stdout:1856: finishing write tid 4 to smithi20438004-6 2024-09-13T11:33:49.942 INFO:tasks.rados.rados.0.smithi204.stdout:1856: oid 6 updating version 0 to 112 2024-09-13T11:33:49.942 INFO:tasks.rados.rados.0.smithi204.stdout:1856: oid 6 version 112 is already newer than 110 2024-09-13T11:33:49.942 INFO:tasks.rados.rados.0.smithi204.stdout:1856: oid 6 version 112 is already newer than 111 2024-09-13T11:33:49.942 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 112 (ObjNum 401 snap 0 seq_num 401) dirty exists 2024-09-13T11:33:49.942 INFO:tasks.rados.rados.0.smithi204.stdout:1856: left oid 6 (ObjNum 401 snap 0 seq_num 401) 2024-09-13T11:33:49.942 INFO:tasks.rados.rados.0.smithi204.stdout:1857: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-13T11:33:49.942 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:49.973 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:49.972+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:49.980 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:49.980 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:49.980 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:49.980 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:49.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:49.999 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 458794 2024-09-13T11:33:50.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-13T11:33:50.092 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:50.092 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:50.092 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:50.092 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:50.092 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:50.098 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:50.096+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:50.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:50.102 INFO:tasks.rados.rados.0.smithi204.stdout:1858: finishing write tid 1 to smithi20438004-39 2024-09-13T11:33:50.102 INFO:tasks.rados.rados.0.smithi204.stdout:1862: got (22) Invalid argument 2024-09-13T11:33:50.102 INFO:tasks.rados.rados.0.smithi204.stdout:1858: finishing write tid 2 to smithi20438004-39 2024-09-13T11:33:50.102 INFO:tasks.rados.rados.0.smithi204.stdout:1859: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-13T11:33:50.123 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1881997 2024-09-13T11:33:50.143 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1471541 2024-09-13T11:33:50.175 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:50.173+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:1858: finishing write tid 3 to smithi20438004-39 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:1858: finishing write tid 4 to smithi20438004-39 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:1858: oid 39 updating version 0 to 135 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:1858: oid 39 updating version 135 to 136 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:1858: oid 39 updating version 136 to 137 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 39 v 137 (ObjNum 402 snap 0 seq_num 402) dirty exists 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:1858: left oid 39 (ObjNum 402 snap 0 seq_num 402) 2024-09-13T11:33:50.179 INFO:tasks.rados.rados.0.smithi204.stdout:1860: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-13T11:33:50.187 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1461221 2024-09-13T11:33:50.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:201 3.1 3:b265d015:::smithi20438004-2 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.121434129999997, 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030631999, 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.209 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 201 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084691+0000", 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084693+0000", 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4739999999999999e-06 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084702+0000", 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.884e-06 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084715+0000", 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2858e-05 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084748+0000", 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3113e-05 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084807+0000", 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8523999999999997e-05 2024-09-13T11:33:50.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084930+0000", 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012317099999999999 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.097750+0000", 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012820575000000001 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107462+0000", 2024-09-13T11:33:50.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097120990000000001 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114842+0000", 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073796520000000004 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115192+0000", 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035007400000000002 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115249+0000", 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7277000000000001e-05 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115323+0000", 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3298000000000002e-05 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:202 3.1 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:50.215 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.121352682999998, 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055756029999999998, 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 202 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084772+0000", 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084773+0000", 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3600000000000003e-07 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084777+0000", 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3540000000000002e-06 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084784+0000", 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084789+0000", 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5480000000000001e-06 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085081+0000", 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000291673 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085197+0000", 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011550800000000001 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.109819+0000", 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024622326999999999 2024-09-13T11:33:50.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115554+0000", 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057346100000000002 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.139296+0000", 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023742025999999999 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140477+0000", 2024-09-13T11:33:50.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011813660000000001 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140507+0000", 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0307e-05 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140528+0000", 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0934e-05 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:203 3.1 3:970ba768:::smithi20438004-4 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.121309914999998, 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.055588820999999997, 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 203 2024-09-13T11:33:50.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084815+0000", 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1900000000000003e-07 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084824+0000", 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3110000000000006e-06 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084831+0000", 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4869999999999996e-06 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084839+0000", 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7840000000000004e-06 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085318+0000", 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047915700000000002 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085404+0000", 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5576999999999997e-05 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.108042+0000", 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022638703999999999 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115550+0000", 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075072150000000002 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132897+0000", 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017347621000000001 2024-09-13T11:33:50.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140260+0000", 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073629139999999999 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140331+0000", 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0994999999999994e-05 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.140404+0000", 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2537e-05 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:206 3.1 3:95cea546:::smithi20438004-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.121274294000003, 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022595217000000001, 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.228 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:50.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084851+0000", 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1799999999999995e-07 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084854+0000", 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1650000000000002e-06 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084861+0000", 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1300000000000003e-06 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084868+0000", 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.55e-06 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085510+0000", 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00064168099999999996 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085595+0000", 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5643999999999995e-05 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.091806+0000", 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062107220000000001 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107236+0000", 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015429897999999999 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107281+0000", 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4994000000000003e-05 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107302+0000", 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1163e-05 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107375+0000", 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2915000000000002e-05 2024-09-13T11:33:50.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107446+0000", 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0837e-05 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:210 3.1 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.121247159000006, 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047980961000000003, 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.234 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 210 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084878+0000", 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084883+0000", 2024-09-13T11:33:50.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084890+0000", 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0650000000000001e-06 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084896+0000", 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9000000000000003e-06 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085703+0000", 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000807097 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085822+0000", 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119098 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.104833+0000", 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019011389 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115543+0000", 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01070981 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127458+0000", 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011915189 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132784+0000", 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053251139999999997 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132825+0000", 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1329999999999999e-05 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132859+0000", 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3704000000000001e-05 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.240 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:215 3.1 3:84f0de12:::smithi20438004-16 low_tier:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.121220842, 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034172806999999999, 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 215 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084904+0000", 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0400000000000002e-07 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084907+0000", 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.666e-06 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084915+0000", 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9330000000000001e-06 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.084920+0000", 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8289999999999998e-06 2024-09-13T11:33:50.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.085942+0000", 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001022029 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.086010+0000", 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8007000000000001e-05 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.100909+0000", 2024-09-13T11:33:50.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014898991 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.107467+0000", 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065583270000000001 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.117910+0000", 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010443309 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.118971+0000", 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010605510000000001 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119023+0000", 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2114000000000003e-05 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.119077+0000", 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3647000000000003e-05 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:218 3.1 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.095538293999994, 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036660768000000003, 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 218 2024-09-13T11:33:50.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110587+0000", 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110588+0000", 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.761e-06 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110603+0000", 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4542000000000001e-05 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110616+0000", 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2795e-05 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.110628+0000", 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1959e-05 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.114882+0000", 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042543379999999999 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000143676 2024-09-13T11:33:50.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.120883+0000", 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058572549999999996 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.123587+0000", 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027042709999999998 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.144002+0000", 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020414505999999999 2024-09-13T11:33:50.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147169+0000", 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031675470000000002 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147204+0000", 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4523999999999997e-05 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147247+0000", 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3594000000000003e-05 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:219 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.095073937999999, 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.054051957999999997, 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.253 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 219 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111051+0000", 2024-09-13T11:33:50.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111052+0000", 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.093e-06 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111059+0000", 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9410000000000001e-06 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111069+0000", 2024-09-13T11:33:50.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0519999999999999e-05 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.111077+0000", 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8290000000000008e-06 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115373+0000", 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042952490000000001 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.115459+0000", 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6367999999999997e-05 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.129610+0000", 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014151285 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.135388+0000", 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057775270000000002 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149276+0000", 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013888773 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165015+0000", 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015738682 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165065+0000", 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0170000000000002e-05 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165103+0000", 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7521000000000002e-05 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:221 3.1 3:baf08732:::smithi20438004-22 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.089681166999995, 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034943881000000003, 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.259 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 221 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116444+0000", 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116445+0000", 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1680000000000001e-06 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116451+0000", 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0379999999999996e-06 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116461+0000", 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0604e-05 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116469+0000", 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8359999999999992e-06 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116516+0000", 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6332999999999998e-05 2024-09-13T11:33:50.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.116579+0000", 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3825999999999995e-05 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.127789+0000", 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011209869000000001 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.132875+0000", 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050855780000000003 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149082+0000", 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016207213000000002 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151312+0000", 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002230358 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151344+0000", 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1081000000000003e-05 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151388+0000", 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3977000000000003e-05 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:230 3.1 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.265 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.059530694000003, 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.046965185999999999, 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146594+0000", 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146596+0000", 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.717e-06 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146606+0000", 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146620+0000", 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4343e-05 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146630+0000", 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3989999999999993e-06 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146705+0000", 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5135999999999993e-05 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146828+0000", 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122714 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165695+0000", 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018866951 2024-09-13T11:33:50.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.171485+0000", 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057905220000000002 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185089+0000", 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013604132999999999 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193461+0000", 2024-09-13T11:33:50.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083717840000000002 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193507+0000", 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5652000000000003e-05 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193559+0000", 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2624000000000003e-05 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.271 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:231 3.1 3:ab075b6a:::smithi20438004-32 low_tier:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.059321342999993, 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033558088, 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 231 2024-09-13T11:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146803+0000", 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146804+0000", 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3300000000000002e-07 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146808+0000", 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4790000000000001e-06 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146814+0000", 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3230000000000003e-06 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146817+0000", 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4769999999999999e-06 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.146955+0000", 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013720299999999999 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.147052+0000", 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7231000000000005e-05 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.156769+0000", 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097173290000000002 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.165122+0000", 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083524589999999996 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.173409+0000", 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082871379999999994 2024-09-13T11:33:50.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180256+0000", 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006846871 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180302+0000", 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6545999999999998e-05 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180362+0000", 2024-09-13T11:33:50.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9499000000000001e-05 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:235 3.1 3:be4a0686:::smithi20438004-36 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.056322917000003, 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035212407000000001, 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 235 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149802+0000", 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149803+0000", 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0580000000000001e-06 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149810+0000", 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149820+0000", 2024-09-13T11:33:50.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0464e-05 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149826+0000", 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149863+0000", 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7416999999999998e-05 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149983+0000", 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011927800000000001 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.160853+0000", 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010870216 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.167925+0000", 2024-09-13T11:33:50.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070717159999999996 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.180224+0000", 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012298972 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184892+0000", 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046685839999999999 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.184959+0000", 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7105000000000002e-05 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185014+0000", 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5099000000000002e-05 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:236 3.1 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [copy-from ver 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.284 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.056209002000003, 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.059325392999999997, 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 236 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149916+0000", 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6599999999999996e-07 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149920+0000", 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149929+0000", 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3769999999999996e-06 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.149937+0000", 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7660000000000004e-06 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150123+0000", 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018567200000000001 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.150254+0000", 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130426 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181070+0000", 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030816856 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.190793+0000", 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097220870000000008 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197985+0000", 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007192431 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209174+0000", 2024-09-13T11:33:50.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011189536999999999 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209209+0000", 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4359000000000001e-05 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.209241+0000", 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2369000000000001e-05 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:237 3.1 3:9a2730a7:::smithi20438004-38 low_tier:head [copy-from ver 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.054256214999995, 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.052738543999999998, 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 237 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151869+0000", 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151870+0000", 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0300000000000001e-06 2024-09-13T11:33:50.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151875+0000", 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2859999999999999e-06 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151890+0000", 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5557999999999999e-05 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151898+0000", 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7500000000000003e-06 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151934+0000", 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5988000000000002e-05 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.151999+0000", 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4394999999999996e-05 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.177811+0000", 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025812831000000001 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185336+0000", 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075243209999999996 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.197387+0000", 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012051345999999999 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204491+0000", 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071034999999999996 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204551+0000", 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0003000000000001e-05 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.204607+0000", 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6536000000000002e-05 2024-09-13T11:33:50.296 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:245 3.1 3:adf25eb4:::smithi20438004-46 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.025067167000003, 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.03651658, 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.297 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 245 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181058+0000", 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181059+0000", 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4780000000000001e-06 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181066+0000", 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5119999999999997e-06 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181078+0000", 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2333e-05 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.181085+0000", 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4449999999999999e-06 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185114+0000", 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040285579999999998 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.185206+0000", 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1935000000000007e-05 2024-09-13T11:33:50.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.195343+0000", 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010137147000000001 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.200849+0000", 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005505874 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.203835+0000", 2024-09-13T11:33:50.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002985705 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217494+0000", 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013658926 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217538+0000", 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4048999999999998e-05 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.217574+0000", 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6618000000000001e-05 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:249 3.1 3:81faf980:::smithi20438004-50 low_tier:head [copy-from ver 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.012148518000004, 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038339094999999997, 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.303 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 249 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193976+0000", 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193977+0000", 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.11e-06 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193984+0000", 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.193995+0000", 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0794e-05 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194002+0000", 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.374e-06 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194061+0000", 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9485999999999999e-05 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.194291+0000", 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000229321 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.222511+0000", 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028219767999999999 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.227779+0000", 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052686319999999997 2024-09-13T11:33:50.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.231941+0000", 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041617809999999998 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232226+0000", 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028545799999999999 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232277+0000", 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0445000000000002e-05 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.232315+0000", 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584999999999997e-05 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:44 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:50.309 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.937724789999997, 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019379767999999999, 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268400+0000", 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268402+0000", 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268410+0000", 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2069999999999995e-06 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268426+0000", 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5588999999999999e-05 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268437+0000", 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0773e-05 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270340+0000", 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001903231 2024-09-13T11:33:50.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270350+0000", 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0543000000000001e-05 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278988+0000", 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086371350000000006 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279079+0000", 2024-09-13T11:33:50.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1012000000000002e-05 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.279189+0000", 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110598 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281564+0000", 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023743929999999998 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281584+0000", 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0655000000000001e-05 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287627+0000", 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060427750000000002 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287716+0000", 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9470999999999998e-05 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287780+0000", 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3329000000000003e-05 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.3.22:45 3.1 3:97e533d4:::smithi20438004-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.937667021999999, 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019566733999999999, 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268458+0000", 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6100000000000001e-07 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268462+0000", 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6430000000000001e-06 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268469+0000", 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7399999999999998e-06 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268475+0000", 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8039999999999998e-06 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270391+0000", 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001916713 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270397+0000", 2024-09-13T11:33:50.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8490000000000002e-06 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280010+0000", 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096124780000000007 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280095+0000", 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5506000000000004e-05 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280235+0000", 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000139405 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281596+0000", 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001361615 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281676+0000", 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0259999999999994e-05 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287969+0000", 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062924000000000001 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.287999+0000", 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0270999999999999e-05 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288025+0000", 2024-09-13T11:33:50.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5488999999999999e-05 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:26 3.1 3:a6c449f0:::2fa16cfaf91e48313b717fb6e16c839aad29b3524900af3b2b4692c59d034ccb:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.937441653, 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019443548000000001, 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268683+0000", 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7500000000000001e-07 2024-09-13T11:33:50.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268686+0000", 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1270000000000001e-06 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268689+0000", 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8709999999999999e-06 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268694+0000", 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4530000000000004e-06 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270498+0000", 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00180423 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270502+0000", 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.625e-06 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.326 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280882+0000", 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010380263000000001 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.280973+0000", 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1199000000000006e-05 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281088+0000", 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011427800000000001 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.327 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282421+0000", 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013333590000000001 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282432+0000", 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.113e-05 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288074+0000", 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056417469999999999 2024-09-13T11:33:50.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288104+0000", 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0402000000000001e-05 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288127+0000", 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2589e-05 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.0.22:28 3.1 3:96eef999:::4534b522f789f408dc72053dbc198d7d54ba3a77b9806bd26ef033725ea1a210:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.937425912999998, 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019504840999999998, 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268699+0000", 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6300000000000001e-07 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268702+0000", 2024-09-13T11:33:50.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.441e-06 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268709+0000", 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.888e-06 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.268714+0000", 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.519e-06 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270524+0000", 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018093740000000001 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.270527+0000", 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.484e-06 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281702+0000", 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011174584 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281828+0000", 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012597699999999999 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,11", 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281975+0000", 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014744000000000001 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282783+0000", 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080754199999999996 2024-09-13T11:33:50.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282862+0000", 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9326999999999995e-05 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288155+0000", 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005293056 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288183+0000", 2024-09-13T11:33:50.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8152000000000001e-05 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.288204+0000", 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0693999999999999e-05 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.336 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:50.339 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:50.339 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1650989 2024-09-13T11:33:50.339 INFO:tasks.rados.rados.0.smithi204.stdout:1861: expect (ObjNum 363 snap 0 seq_num 363) 2024-09-13T11:33:50.339 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1149655 2024-09-13T11:33:50.339 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:50.339 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:33:50.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:50.340 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:50.336+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:50.343 INFO:tasks.rados.rados.0.smithi204.stdout:1863: got (0) Success 2024-09-13T11:33:50.343 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 109 (ObjNum 396 snap 0 seq_num 396) dirty exists 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1856: done (7 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1857: done (6 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1858: done (5 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1859: done (4 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1860: done (3 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1861: done (2 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1862: done (1 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1863: done (0 left) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1864: read oid 24 snap -1 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1864: expect (ObjNum 363 snap 0 seq_num 363) 2024-09-13T11:33:50.344 INFO:tasks.rados.rados.0.smithi204.stdout:1865: set_chunk oid 21 target oid 2024-09-13T11:33:50.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:50.438 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:50.437+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:50.444 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:50.444 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:50.444 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:33:50.444 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:50.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:50.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-13T11:33:50.540 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:50.539+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:50.558 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:50.558 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:33:50.558 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:33:50.558 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:33:50.558 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:50.561 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:50.561 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:50.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:50.596 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:50.596 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:50.596 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:50.613 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:50.614 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:50.630 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:50.631 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:50.641 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:50.639+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:50.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2024-09-13T11:33:50.647 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:50.647 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:50.665 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:50.665 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:50.672 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:33:50.672 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:33:50.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:33:50.672 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:33:50.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.672 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:254 4.0 4:04aeeab9:::smithi20438004-5:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.673 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:50.673 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.468458607000002, 2024-09-13T11:33:50.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022108323999999999, 2024-09-13T11:33:50.673 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.673 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.673 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.673 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 254 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:33:50.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201594+0000", 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201603+0000", 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.568e-06 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201619+0000", 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6540000000000001e-05 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201629+0000", 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3200000000000006e-06 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201685+0000", 2024-09-13T11:33:50.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6078000000000001e-05 2024-09-13T11:33:50.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201747+0000", 2024-09-13T11:33:50.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2242999999999993e-05 2024-09-13T11:33:50.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:50.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220103+0000", 2024-09-13T11:33:50.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01835548 2024-09-13T11:33:50.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220937+0000", 2024-09-13T11:33:50.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083429299999999995 2024-09-13T11:33:50.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221938+0000", 2024-09-13T11:33:50.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010010710000000001 2024-09-13T11:33:50.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223567+0000", 2024-09-13T11:33:50.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001628846 2024-09-13T11:33:50.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223638+0000", 2024-09-13T11:33:50.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1165000000000001e-05 2024-09-13T11:33:50.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223701+0000", 2024-09-13T11:33:50.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3120999999999994e-05 2024-09-13T11:33:50.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.683 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.684 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:285 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:50.684 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:50.684 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.435474147999997, 2024-09-13T11:33:50.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012191677, 2024-09-13T11:33:50.684 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.684 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.685 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.685 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.685 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.685 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.685 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 285 2024-09-13T11:33:50.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.686 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:50.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:33:50.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234578+0000", 2024-09-13T11:33:50.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:33:50.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234583+0000", 2024-09-13T11:33:50.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9110000000000001e-06 2024-09-13T11:33:50.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234593+0000", 2024-09-13T11:33:50.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:33:50.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234598+0000", 2024-09-13T11:33:50.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9509999999999997e-06 2024-09-13T11:33:50.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234639+0000", 2024-09-13T11:33:50.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1826999999999998e-05 2024-09-13T11:33:50.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234688+0000", 2024-09-13T11:33:50.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9039000000000003e-05 2024-09-13T11:33:50.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:33:50.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239203+0000", 2024-09-13T11:33:50.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045143739999999998 2024-09-13T11:33:50.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239979+0000", 2024-09-13T11:33:50.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077664100000000001 2024-09-13T11:33:50.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.242634+0000", 2024-09-13T11:33:50.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026544899999999998 2024-09-13T11:33:50.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246683+0000", 2024-09-13T11:33:50.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040491499999999996 2024-09-13T11:33:50.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246723+0000", 2024-09-13T11:33:50.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0429000000000002e-05 2024-09-13T11:33:50.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246769+0000", 2024-09-13T11:33:50.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5430999999999999e-05 2024-09-13T11:33:50.700 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.700 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.700 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.701 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1715 4.f 4:fea8c5b2:::smithi20438004-39:head [write 769787~704463,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:50.701 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:50.701 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.401665207999997, 2024-09-13T11:33:50.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020210576000000001, 2024-09-13T11:33:50.702 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.702 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.702 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.703 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1715 2024-09-13T11:33:50.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.703 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:50.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:33:50.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268388+0000", 2024-09-13T11:33:50.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:33:50.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269172+0000", 2024-09-13T11:33:50.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078478 2024-09-13T11:33:50.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269184+0000", 2024-09-13T11:33:50.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.182e-05 2024-09-13T11:33:50.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269194+0000", 2024-09-13T11:33:50.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8570000000000007e-06 2024-09-13T11:33:50.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269269+0000", 2024-09-13T11:33:50.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4795000000000005e-05 2024-09-13T11:33:50.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269283+0000", 2024-09-13T11:33:50.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:33:50.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275426+0000", 2024-09-13T11:33:50.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061428050000000003 2024-09-13T11:33:50.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275540+0000", 2024-09-13T11:33:50.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011432199999999999 2024-09-13T11:33:50.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:50.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275658+0000", 2024-09-13T11:33:50.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118071 2024-09-13T11:33:50.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278252+0000", 2024-09-13T11:33:50.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594224 2024-09-13T11:33:50.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279142+0000", 2024-09-13T11:33:50.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088964600000000001 2024-09-13T11:33:50.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288321+0000", 2024-09-13T11:33:50.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091785600000000005 2024-09-13T11:33:50.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288363+0000", 2024-09-13T11:33:50.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2957000000000002e-05 2024-09-13T11:33:50.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288492+0000", 2024-09-13T11:33:50.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012896899999999999 2024-09-13T11:33:50.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288597+0000", 2024-09-13T11:33:50.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010429000000000001 2024-09-13T11:33:50.718 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.718 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.718 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.719 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1716 4.f 4:fea8c5b2:::smithi20438004-39:head [write 2267961~101826] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:50.719 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:50.719 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.400838351000004, 2024-09-13T11:33:50.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019610079999999998, 2024-09-13T11:33:50.719 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.720 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.720 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.720 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.720 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1716 2024-09-13T11:33:50.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.721 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:50.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:33:50.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269214+0000", 2024-09-13T11:33:50.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9699999999999995e-07 2024-09-13T11:33:50.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269305+0000", 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1330999999999997e-05 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269314+0000", 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9209999999999998e-06 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269321+0000", 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269363+0000", 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2388999999999997e-05 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269372+0000", 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277175+0000", 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078032580000000004 2024-09-13T11:33:50.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277247+0000", 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1773999999999994e-05 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277354+0000", 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010704000000000001 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288308+0000", 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010953353000000001 2024-09-13T11:33:50.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288328+0000", 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0353e-05 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288640+0000", 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031178999999999999 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288706+0000", 2024-09-13T11:33:50.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6749999999999996e-05 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288765+0000", 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8876999999999999e-05 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288823+0000", 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7806999999999997e-05 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1717 4.f 4:fea8c5b2:::smithi20438004-39:head [setxattr _header (56) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.400714089000004, 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019744788999999999, 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1717 2024-09-13T11:33:50.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269338+0000", 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269342+0000", 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.924e-06 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269349+0000", 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9619999999999996e-06 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269355+0000", 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1210000000000003e-06 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269399+0000", 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4360000000000002e-05 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5570000000000001e-06 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278293+0000", 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088867549999999997 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278372+0000", 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9561999999999995e-05 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278492+0000", 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119654 2024-09-13T11:33:50.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288335+0000", 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098426840000000008 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288672+0000", 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033785899999999998 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288863+0000", 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019105599999999999 2024-09-13T11:33:50.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288904+0000", 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0738000000000002e-05 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288949+0000", 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4443000000000001e-05 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289082+0000", 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013346199999999999 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1718 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.400684615000003, 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019946973, 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.736 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1718 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:50.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8999999999999996e-07 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269373+0000", 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8629999999999996e-06 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269379+0000", 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4979999999999997e-06 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269383+0000", 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7260000000000002e-06 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269443+0000", 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9616e-05 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269451+0000", 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279055+0000", 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009604586 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279104+0000", 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8131999999999999e-05 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289119+0000", 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010015616 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289177+0000", 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8060999999999999e-05 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289314+0000", 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000136525 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1732 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.400660119999998, 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017069699000000001, 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.742 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1732 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269392+0000", 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269395+0000", 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-06 2024-09-13T11:33:50.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269400+0000", 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.198e-06 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9410000000000004e-06 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269514+0000", 2024-09-13T11:33:50.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108124 2024-09-13T11:33:50.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269524+0000", 2024-09-13T11:33:50.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:33:50.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279170+0000", 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096457209999999995 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279248+0000", 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8147999999999998e-05 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286461+0000", 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072129789999999996 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.749 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1733 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.400617143999995, 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0187283, 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1733 2024-09-13T11:33:50.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:33:50.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269435+0000", 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7800000000000001e-07 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269438+0000", 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.535e-06 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269447+0000", 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6559999999999998e-06 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269452+0000", 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.972e-06 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269552+0000", 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9820000000000003e-05 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269556+0000", 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286496+0000", 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016940182000000002 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286550+0000", 2024-09-13T11:33:50.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3640999999999997e-05 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288163+0000", 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016126720000000001 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1734 4.f 4:f653a832:::smithi20438004-19:head [read 1042153~2571319,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.400551242999995, 2024-09-13T11:33:50.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021098286000000001, 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1734 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269501+0000", 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269504+0000", 2024-09-13T11:33:50.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.58e-06 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269511+0000", 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6200000000000001e-06 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269517+0000", 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7030000000000003e-06 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269580+0000", 2024-09-13T11:33:50.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269583+0000", 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5020000000000001e-06 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289341+0000", 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019757429 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289385+0000", 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3939000000000001e-05 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.290598+0000", 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012136130000000001 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.760 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.12.0:167 4.6 e51/31 4:6978b14d:::smithi20438004-23:head v 51'52, mlcod=45'50)", 2024-09-13T11:33:50.761 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:50.761 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 63.478436899999998, 2024-09-13T11:33:50.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013003298, 2024-09-13T11:33:50.761 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.761 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.761 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.762 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:50.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:33:50.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191616+0000", 2024-09-13T11:33:50.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0270000000000001e-06 2024-09-13T11:33:50.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196708+0000", 2024-09-13T11:33:50.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050925720000000001 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196728+0000", 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9616999999999999e-05 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196738+0000", 2024-09-13T11:33:50.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8330000000000006e-06 2024-09-13T11:33:50.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196817+0000", 2024-09-13T11:33:50.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9225000000000001e-05 2024-09-13T11:33:50.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196841+0000", 2024-09-13T11:33:50.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3972e-05 2024-09-13T11:33:50.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204573+0000", 2024-09-13T11:33:50.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077323629999999999 2024-09-13T11:33:50.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204618+0000", 2024-09-13T11:33:50.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4688999999999999e-05 2024-09-13T11:33:50.769 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.769 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.769 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1770 4.0 4:001d90c2:::smithi20438004-28:head [read 0~2218524,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 63.400262282, 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011933050000000001, 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.770 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1770 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269789+0000", 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269791+0000", 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4300000000000001e-06 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269795+0000", 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269803+0000", 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-13T11:33:50.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269810+0000", 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.246e-06 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273037+0000", 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032276700000000002 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273048+0000", 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0511e-05 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.277997+0000", 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049497539999999998 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.278150+0000", 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152406 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.281722+0000", 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00357224 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1771 4.0 4:001d90c2:::smithi20438004-28:head [read 633380~1585144,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 63.400232277999997, 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013022562, 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.775 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1771 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:33:50.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269820+0000", 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4000000000000002e-07 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269821+0000", 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9269999999999999e-06 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269825+0000", 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8789999999999996e-06 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269829+0000", 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4740000000000001e-06 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273144+0000", 2024-09-13T11:33:50.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033155459999999999 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273155+0000", 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.062e-05 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282280+0000", 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091252620000000003 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282403+0000", 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012240499999999999 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282842+0000", 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043900900000000002 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1772 4.0 4:001d90c2:::smithi20438004-28:head [sparse-read 2038646~179878] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:50.780 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 63.400218832999997, 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013177762000000001, 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1772 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9499999999999998e-07 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269834+0000", 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269837+0000", 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.005e-06 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269843+0000", 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6389999999999997e-06 2024-09-13T11:33:50.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273188+0000", 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033454840000000001 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273196+0000", 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0830000000000004e-06 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282874+0000", 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096776810000000005 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282924+0000", 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0095000000000001e-05 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.283010+0000", 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6216000000000003e-05 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2324 4.16 4:6978b14d:::smithi20438004-23:head [write 544593~799729,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.532970851999998, 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14427186, 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2324 2024-09-13T11:33:50.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7100000000000001e-07 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137660+0000", 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057879000000000003 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137671+0000", 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0518e-05 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137675+0000", 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137754+0000", 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8492000000000007e-05 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137765+0000", 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1027e-05 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273433+0000", 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13566771999999999 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273569+0000", 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013616999999999999 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:50.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273699+0000", 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130142 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281227+0000", 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075284699999999998 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281259+0000", 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1610999999999999e-05 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281298+0000", 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9138999999999999e-05 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281352+0000", 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.792 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2325 4.16 4:6978b14d:::smithi20438004-23:head [write 1761799~776065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.526305536000002, 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13997515799999999, 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2325 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143748+0000", 2024-09-13T11:33:50.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000001e-06 2024-09-13T11:33:50.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144385+0000", 2024-09-13T11:33:50.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000637125 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144396+0000", 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0633e-05 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144401+0000", 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2469999999999999e-06 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144457+0000", 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5566999999999998e-05 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144467+0000", 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8819999999999996e-06 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275371+0000", 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13090408000000001 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275440+0000", 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8749000000000002e-05 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275530+0000", 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0236999999999997e-05 2024-09-13T11:33:50.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282215+0000", 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066849240000000001 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283575+0000", 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001360775 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283629+0000", 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283721+0000", 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1799999999999995e-05 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2326 4.16 4:6978b14d:::smithi20438004-23:head [write 2948352~396241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.521940115, 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13583619799999999, 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.801 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2326 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:33:50.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148113+0000", 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148305+0000", 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000192008 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148316+0000", 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0293e-05 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148327+0000", 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1489e-05 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148398+0000", 2024-09-13T11:33:50.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0421999999999998e-05 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148407+0000", 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1090000000000004e-06 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279356+0000", 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13094884700000001 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279444+0000", 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279534+0000", 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9573000000000005e-05 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282581+0000", 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030473990000000001 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283856+0000", 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001275307 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283901+0000", 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5238999999999998e-05 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283947+0000", 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5972e-05 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2327 4.16 4:6978b14d:::smithi20438004-23:head [setxattr _header (56) in=7b,truncate 3344593] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.521712614999998, 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13580746099999999, 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.808 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2327 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.34e-07 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148341+0000", 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0999999999999998e-06 2024-09-13T11:33:50.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148363+0000", 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2045999999999999e-05 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148369+0000", 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.665e-06 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148463+0000", 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4475999999999995e-05 2024-09-13T11:33:50.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148474+0000", 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0319999999999999e-05 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280931+0000", 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132457142 2024-09-13T11:33:50.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280994+0000", 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2693000000000004e-05 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281088+0000", 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3993000000000006e-05 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:33:50.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282632+0000", 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001544199 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283989+0000", 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013571670000000001 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284048+0000", 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9070999999999997e-05 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284146+0000", 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8255e-05 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2328 4.16 4:6978b14d:::smithi20438004-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 54.521678100000003, 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13592306100000001, 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2328 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148374+0000", 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6899999999999999e-07 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148375+0000", 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148378+0000", 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0989999999999999e-06 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148381+0000", 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148500+0000", 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119052 2024-09-13T11:33:50.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148509+0000", 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8759999999999994e-06 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281377+0000", 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132868244 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281422+0000", 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4909e-05 2024-09-13T11:33:50.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284171+0000", 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027497300000000001 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284216+0000", 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4626999999999998e-05 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284296+0000", 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0275e-05 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2687 4.10 4:0c9026a3:::smithi20438004-36:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 44.598225915999997, 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029783952999999998, 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.821 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2687 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071827+0000", 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.096e-06 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071831+0000", 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8210000000000001e-06 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071843+0000", 2024-09-13T11:33:50.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1715e-05 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071848+0000", 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071899+0000", 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0290000000000001e-05 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071988+0000", 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.101609+0000", 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0296212 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.825 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.826 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3509 4.f 4:f5aacf61:::smithi20438004-27:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:50.826 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:50.826 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 26.890706738999999, 2024-09-13T11:33:50.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022491955000000001, 2024-09-13T11:33:50.827 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:33:50.827 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:33:50.827 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:33:50.827 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:33:50.827 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:33:50.827 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:50.827 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3509 2024-09-13T11:33:50.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.828 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:33:50.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:33:50.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:50.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:33:50.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:33:50.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:33:50.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:33:50.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779347+0000", 2024-09-13T11:33:50.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9209999999999999e-06 2024-09-13T11:33:50.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:33:50.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779354+0000", 2024-09-13T11:33:50.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7980000000000006e-06 2024-09-13T11:33:50.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:33:50.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779373+0000", 2024-09-13T11:33:50.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289999999999999e-05 2024-09-13T11:33:50.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:33:50.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779399+0000", 2024-09-13T11:33:50.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:50.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:33:50.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779457+0000", 2024-09-13T11:33:50.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8888000000000003e-05 2024-09-13T11:33:50.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:33:50.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779523+0000", 2024-09-13T11:33:50.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5995999999999996e-05 2024-09-13T11:33:50.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:33:50.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:33:50.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:33:50.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.801837+0000", 2024-09-13T11:33:50.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022313116000000001 2024-09-13T11:33:50.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.836 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:33:50.836 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:33:50.836 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:33:50.838 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:50.839 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:50.840 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:50.840 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:50.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:50.841 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:50.837+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:50.842 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:50.842 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:50.843 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:50.843 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:50.844 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:50.845 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:50.845 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:50.845 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:50.846 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:50.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2024-09-13T11:33:50.848 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:50.848 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:50.849 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:50.850 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:50.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:50.939 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:50.937+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:50.984 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:50.984 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:50.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:51.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2024-09-13T11:33:51.039 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:51.039+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:51.096 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:51.096 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:51.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:51.148 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:51.147+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:51.208 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:51.209 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:51.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 reset_purged_snaps_last 2024-09-13T11:33:51.216 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:51.241 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:51.241+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:51.310 INFO:tasks.rados.rados.0.smithi204.stdout:1865: set_chunk oid 21 offset: 590848 length: 1398272 target oid 56fc55f1d5973d4628acc2304c554556b78e3d44591049a058f15c9de223279a offset: 0 snap -1 2024-09-13T11:33:51.310 INFO:tasks.rados.rados.0.smithi204.stdout:1866: write oid 45 current snap is 0 2024-09-13T11:33:51.311 INFO:tasks.rados.rados.0.smithi204.stdout:1866: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-13T11:33:51.317 INFO:tasks.rados.rados.0.smithi204.stdout:1866: writing smithi20438004-45 from 773221 to 1265277 tid 1 2024-09-13T11:33:51.323 INFO:tasks.rados.rados.0.smithi204.stdout:1866: writing smithi20438004-45 from 1816008 to 2319134 tid 2 2024-09-13T11:33:51.323 INFO:tasks.rados.rados.0.smithi204.stdout:1867: read oid 40 snap -1 2024-09-13T11:33:51.323 INFO:tasks.rados.rados.0.smithi204.stdout:1867: expect deleted 2024-09-13T11:33:51.323 INFO:tasks.rados.rados.0.smithi204.stdout:1868: delete oid 35 current snap is 0 2024-09-13T11:33:51.323 INFO:tasks.rados.rados.0.smithi204.stdout:1864: expect (ObjNum 363 snap 0 seq_num 363) 2024-09-13T11:33:51.325 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:51.329 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:51.329 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-09-13T11:33:51.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 reset_purged_snaps_last 2024-09-13T11:33:51.338 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:51.338 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:51.343 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:51.341+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:51.347 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:51.375 INFO:tasks.rados.rados.0.smithi204.stdout:1865: finishing set_chunk to oid 21 2024-09-13T11:33:51.375 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 41 (ObjNum 372 snap 0 seq_num 372) dirty exists 2024-09-13T11:33:51.375 INFO:tasks.rados.rados.0.smithi204.stdout:1866: finishing write tid 1 to smithi20438004-45 2024-09-13T11:33:51.375 INFO:tasks.rados.rados.0.smithi204.stdout:1866: finishing write tid 2 to smithi20438004-45 2024-09-13T11:33:51.375 INFO:tasks.rados.rados.0.smithi204.stdout:1864: done (4 left) 2024-09-13T11:33:51.375 INFO:tasks.rados.rados.0.smithi204.stdout:1865: done (3 left) 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1867: done (2 left) 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1868: done (1 left) 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1869: tier_evict oid 38 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1869: tier_evict oid 38 snap -1 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1866: finishing write tid 3 to smithi20438004-45 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1866: finishing write tid 4 to smithi20438004-45 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1866: oid 45 updating version 0 to 87 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1866: oid 45 updating version 87 to 88 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1866: oid 45 version 88 is already newer than 86 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 88 (ObjNum 403 snap 0 seq_num 403) dirty exists 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1866: left oid 45 (ObjNum 403 snap 0 seq_num 403) 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1866: done (1 left) 2024-09-13T11:33:51.376 INFO:tasks.rados.rados.0.smithi204.stdout:1870: write (excl) oid 44 current snap is 0 2024-09-13T11:33:51.377 INFO:tasks.rados.rados.0.smithi204.stdout:1870: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-13T11:33:51.380 INFO:tasks.rados.rados.0.smithi204.stdout:1870: writing smithi20438004-44 from 406835 to 1049389 tid 1 2024-09-13T11:33:51.385 INFO:tasks.rados.rados.0.smithi204.stdout:1870: writing smithi20438004-44 from 1532972 to 2039767 tid 2 2024-09-13T11:33:51.390 INFO:tasks.rados.rados.0.smithi204.stdout:1870: writing smithi20438004-44 from 2772312 to 2806835 tid 3 2024-09-13T11:33:51.390 INFO:tasks.rados.rados.0.smithi204.stdout:1871: read oid 39 snap -1 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1871: expect (ObjNum 402 snap 0 seq_num 402) 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1869: got (22) Invalid argument 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1870: finishing write tid 1 to smithi20438004-44 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1870: finishing write tid 2 to smithi20438004-44 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1872: read oid 25 snap -1 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1872: expect (ObjNum 377 snap 0 seq_num 377) 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1869: done (3 left) 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1873: read oid 30 snap -1 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1873: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1874: read oid 3 snap -1 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1874: expect (ObjNum 329 snap 0 seq_num 329) 2024-09-13T11:33:51.391 INFO:tasks.rados.rados.0.smithi204.stdout:1875: write (excl) oid 5 current snap is 0 2024-09-13T11:33:51.392 INFO:tasks.rados.rados.0.smithi204.stdout:1875: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-13T11:33:51.396 INFO:tasks.rados.rados.0.smithi204.stdout:1875: writing smithi20438004-5 from 423642 to 1141445 tid 1 2024-09-13T11:33:51.405 INFO:tasks.rados.rados.0.smithi204.stdout:1875: writing smithi20438004-5 from 1875101 to 2641907 tid 2 2024-09-13T11:33:51.405 INFO:tasks.rados.rados.0.smithi204.stdout:1876: read oid 28 snap -1 2024-09-13T11:33:51.405 INFO:tasks.rados.rados.0.smithi204.stdout:1876: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-13T11:33:51.405 INFO:tasks.rados.rados.0.smithi204.stdout:1877: read oid 49 snap -1 2024-09-13T11:33:51.405 INFO:tasks.rados.rados.0.smithi204.stdout:1877: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1878: read oid 20 snap -1 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1878: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1879: tier_promote oid 10 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1870: finishing write tid 3 to smithi20438004-44 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1880: read oid 14 snap -1 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1880: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1881: write (excl) oid 45 current snap is 0 2024-09-13T11:33:51.406 INFO:tasks.rados.rados.0.smithi204.stdout:1872: expect (ObjNum 377 snap 0 seq_num 377) 2024-09-13T11:33:51.444 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:51.442+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:51.449 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 176093 2024-09-13T11:33:51.465 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:51.465 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:51.474 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:51.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 reset_purged_snaps_last 2024-09-13T11:33:51.541 INFO:tasks.rados.rados.0.smithi204.stdout:1873: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-13T11:33:51.561 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:51.559+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:51.619 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:51.619 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:51.627 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:51.657 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:51.657 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:51.656+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:51.718 INFO:tasks.rados.rados.0.smithi204.stdout:1870: finishing write tid 4 to smithi20438004-44 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1870: finishing write tid 5 to smithi20438004-44 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1870: oid 44 updating version 0 to 85 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1870: oid 44 version 85 is already newer than 83 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1870: oid 44 version 85 is already newer than 84 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1870: oid 44 updating version 85 to 86 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 86 (ObjNum 405 snap 0 seq_num 405) dirty exists 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1870: left oid 44 (ObjNum 405 snap 0 seq_num 405) 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1879: got (0) Success 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 126 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1875: finishing write tid 1 to smithi20438004-5 2024-09-13T11:33:51.719 INFO:tasks.rados.rados.0.smithi204.stdout:1881: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-13T11:33:51.725 INFO:tasks.rados.rados.0.smithi204.stdout:1881: writing smithi20438004-45 from 440449 to 1233501 tid 1 2024-09-13T11:33:51.732 INFO:tasks.rados.rados.0.smithi204.stdout:1881: writing smithi20438004-45 from 1933583 to 2444047 tid 2 2024-09-13T11:33:51.732 INFO:tasks.rados.rados.0.smithi204.stdout:1870: done (11 left) 2024-09-13T11:33:51.732 INFO:tasks.rados.rados.0.smithi204.stdout:1872: done (10 left) 2024-09-13T11:33:51.732 INFO:tasks.rados.rados.0.smithi204.stdout:1873: done (9 left) 2024-09-13T11:33:51.732 INFO:tasks.rados.rados.0.smithi204.stdout:1879: done (8 left) 2024-09-13T11:33:51.732 INFO:tasks.rados.rados.0.smithi204.stdout:1882: read oid 37 snap -1 2024-09-13T11:33:51.733 INFO:tasks.rados.rados.0.smithi204.stdout:1882: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:51.733 INFO:tasks.rados.rados.0.smithi204.stdout:1883: write oid 19 current snap is 0 2024-09-13T11:33:51.733 INFO:tasks.rados.rados.0.smithi204.stdout:1875: finishing write tid 2 to smithi20438004-5 2024-09-13T11:33:51.733 INFO:tasks.rados.rados.0.smithi204.stdout:1883: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-13T11:33:51.737 INFO:tasks.rados.rados.0.smithi204.stdout:1883: writing smithi20438004-19 from 457256 to 925557 tid 1 2024-09-13T11:33:51.745 INFO:tasks.rados.rados.0.smithi204.stdout:1883: writing smithi20438004-19 from 1592065 to 2362540 tid 2 2024-09-13T11:33:51.746 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:51.745+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:51.748 INFO:tasks.rados.rados.0.smithi204.stdout:1883: writing smithi20438004-19 from 2854581 to 2857256 tid 3 2024-09-13T11:33:51.748 INFO:tasks.rados.rados.0.smithi204.stdout:1884: set_chunk oid 6 target oid 2024-09-13T11:33:51.755 INFO:tasks.rados.rados.0.smithi204.stdout:1884: set_chunk oid 6 offset: 1493504 length: 883712 target oid 3a2d1eda74f0a75fde190f0cd2d016950ce32526a1db437307e69a12ecf7112f offset: 0 snap -1 2024-09-13T11:33:51.755 INFO:tasks.rados.rados.0.smithi204.stdout:1885: read oid 50 snap -1 2024-09-13T11:33:51.755 INFO:tasks.rados.rados.0.smithi204.stdout:1885: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-13T11:33:51.755 INFO:tasks.rados.rados.0.smithi204.stdout:1886: read oid 13 snap -1 2024-09-13T11:33:51.756 INFO:tasks.rados.rados.0.smithi204.stdout:1886: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:51.756 INFO:tasks.rados.rados.0.smithi204.stdout:1887: set_chunk oid 31 target oid 2024-09-13T11:33:51.768 INFO:tasks.rados.rados.0.smithi204.stdout:1887: set_chunk oid 31 offset: 657408 length: 1502720 target oid 43b0be01b665b6b51d8574b89b0731ed736905123ee0460866489d47e9afc069 offset: 0 snap -1 2024-09-13T11:33:51.769 INFO:tasks.rados.rados.0.smithi204.stdout:1888: set_chunk oid 7 target oid 2024-09-13T11:33:51.769 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:51.769 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:51.780 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:51.792 INFO:tasks.rados.rados.0.smithi204.stdout:1888: set_chunk oid 7 offset: 69120 length: 2770944 target oid bbdc53b9adb7d018b4a6315886ca29076a5aba5335d59449da8c448c0e02d59e offset: 0 snap -1 2024-09-13T11:33:51.792 INFO:tasks.rados.rados.0.smithi204.stdout:1889: read oid 27 snap -1 2024-09-13T11:33:51.792 INFO:tasks.rados.rados.0.smithi204.stdout:1889: expect (ObjNum 391 snap 0 seq_num 391) 2024-09-13T11:33:51.792 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:51.792 INFO:tasks.rados.rados.0.smithi204.stdout:1875: finishing write tid 3 to smithi20438004-5 2024-09-13T11:33:51.792 INFO:tasks.rados.rados.0.smithi204.stdout:1875: finishing write tid 4 to smithi20438004-5 2024-09-13T11:33:51.792 INFO:tasks.rados.rados.0.smithi204.stdout:1875: oid 5 updating version 0 to 93 2024-09-13T11:33:51.793 INFO:tasks.rados.rados.0.smithi204.stdout:1875: oid 5 updating version 93 to 94 2024-09-13T11:33:51.793 INFO:tasks.rados.rados.0.smithi204.stdout:1875: oid 5 version 94 is already newer than 92 2024-09-13T11:33:51.793 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 5 v 94 (ObjNum 406 snap 0 seq_num 406) dirty exists 2024-09-13T11:33:51.793 INFO:tasks.rados.rados.0.smithi204.stdout:1875: left oid 5 (ObjNum 406 snap 0 seq_num 406) 2024-09-13T11:33:51.793 INFO:tasks.rados.rados.0.smithi204.stdout:1876: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-13T11:33:51.793 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:51.845 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1059043 2024-09-13T11:33:51.847 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:51.846+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:51.885 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:51.885 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:51.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:51.922 INFO:tasks.rados.rados.0.smithi204.stdout:1880: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-13T11:33:51.922 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:51.948 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:51.947+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:52.025 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.025 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.029 INFO:tasks.rados.rados.0.smithi204.stdout:1871: expect (ObjNum 402 snap 0 seq_num 402) 2024-09-13T11:33:52.033 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:52.049 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:52.047+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:52.085 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:52.137 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.137 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.139 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:52.145 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:52.150 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:52.149+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:52.195 INFO:tasks.rados.rados.0.smithi204.stdout:1877: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:52.215 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2165204 2024-09-13T11:33:52.222 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:52.249 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.249 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.250 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:52.249+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:52.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:52.263 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 480056 2024-09-13T11:33:52.297 INFO:tasks.rados.rados.0.smithi204.stdout:1874: expect (ObjNum 329 snap 0 seq_num 329) 2024-09-13T11:33:52.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:33:52.363 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.363 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.371 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:52.378 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:52.386 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:52.384+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:52.451 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:52.451+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:52.482 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.482 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.491 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:52.542 INFO:tasks.rados.rados.0.smithi204.stdout:1878: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-13T11:33:52.542 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:52.552 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:52.551+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:52.590 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1255561 2024-09-13T11:33:52.595 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.595 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.603 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:52.638 INFO:tasks.rados.rados.0.smithi204.stdout:1881: finishing write tid 1 to smithi20438004-45 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1871: done (15 left) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1874: done (14 left) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1875: done (13 left) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1876: done (12 left) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1877: done (11 left) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1878: done (10 left) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1880: done (9 left) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1890: read oid 43 snap -1 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1890: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1891: read oid 10 snap -1 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1891: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-13T11:33:52.639 INFO:tasks.rados.rados.0.smithi204.stdout:1892: read oid 49 snap -1 2024-09-13T11:33:52.640 INFO:tasks.rados.rados.0.smithi204.stdout:1892: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:52.640 INFO:tasks.rados.rados.0.smithi204.stdout:1893: read oid 11 snap -1 2024-09-13T11:33:52.640 INFO:tasks.rados.rados.0.smithi204.stdout:1893: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:52.640 INFO:tasks.rados.rados.0.smithi204.stdout:1894: delete oid 3 current snap is 0 2024-09-13T11:33:52.640 INFO:tasks.rados.rados.0.smithi204.stdout:1881: finishing write tid 2 to smithi20438004-45 2024-09-13T11:33:52.640 INFO:tasks.rados.rados.0.smithi204.stdout:1882: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:52.640 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1293883 2024-09-13T11:33:52.653 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:52.652+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:1881: finishing write tid 3 to smithi20438004-45 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:1881: finishing write tid 4 to smithi20438004-45 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:1881: oid 45 updating version 0 to 89 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:1881: oid 45 updating version 89 to 91 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:1881: oid 45 version 91 is already newer than 90 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 91 (ObjNum 407 snap 0 seq_num 407) dirty exists 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:1881: left oid 45 (ObjNum 407 snap 0 seq_num 407) 2024-09-13T11:33:52.699 INFO:tasks.rados.rados.0.smithi204.stdout:1883: finishing write tid 1 to smithi20438004-19 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: finishing write tid 2 to smithi20438004-19 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: finishing write tid 3 to smithi20438004-19 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: finishing write tid 4 to smithi20438004-19 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: finishing write tid 5 to smithi20438004-19 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: oid 19 updating version 0 to 136 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: oid 19 updating version 136 to 138 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: oid 19 version 138 is already newer than 135 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: oid 19 version 138 is already newer than 137 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 138 (ObjNum 408 snap 0 seq_num 408) dirty exists 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1883: left oid 19 (ObjNum 408 snap 0 seq_num 408) 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1884: finishing set_chunk to oid 6 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 6 v 112 (ObjNum 401 snap 0 seq_num 401) dirty exists 2024-09-13T11:33:52.700 INFO:tasks.rados.rados.0.smithi204.stdout:1885: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-13T11:33:52.710 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.710 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:52.735 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 344524 2024-09-13T11:33:52.753 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:52.752+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:52.781 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:52.827 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.827 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.828 INFO:tasks.rados.rados.0.smithi204.stdout:1887: finishing set_chunk to oid 31 2024-09-13T11:33:52.828 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk returned error code -95 offset: 657408 length: 1502720 2024-09-13T11:33:52.828 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-13T11:33:52.829 INFO:tasks.rados.rados.0.smithi204.stdout:1886: expect (ObjNum 211 snap 0 seq_num 211) 2024-09-13T11:33:52.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:33:52.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:52.879 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:52.877+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:52.943 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:52.943 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:52.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:33:52.956 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:52.954+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:53.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:53.005 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:53.024 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:53.042 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:53.042 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:53.056 INFO:tasks.rados.rados.0.smithi204.stdout:1888: finishing set_chunk to oid 7 2024-09-13T11:33:53.056 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 69120 length: 2770944 2024-09-13T11:33:53.056 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-13T11:33:53.056 INFO:tasks.rados.rados.0.smithi204.stdout:1889: expect (ObjNum 391 snap 0 seq_num 391) 2024-09-13T11:33:53.057 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:53.054+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:53.059 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:53.059 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:53.062 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 761581 2024-09-13T11:33:53.064 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 107.324564022, 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:33:53.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:53.069 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.584638394999999, 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:33:53.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.579579836999997, 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.082 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.579156014999995, 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:53.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:33:53.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:33:53.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.579137970999994, 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:33:53.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:33:53.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:33:53.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 85.862897199000003, 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.094 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:33:53.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:33:53.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:33:53.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 85.849056410000003, 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:33:53.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:607 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 85.750088435999999, 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063619109999999996, 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.106 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 607 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311655+0000", 2024-09-13T11:33:53.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311656+0000", 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311661+0000", 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9309999999999999e-06 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311671+0000", 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8630000000000003e-06 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311676+0000", 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4720000000000003e-06 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311729+0000", 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3071000000000001e-05 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.311809+0000", 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9239999999999993e-05 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316314+0000", 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045057689999999997 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.316772+0000", 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045782199999999998 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317147+0000", 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037501400000000001 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317962+0000", 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081491699999999996 2024-09-13T11:33:53.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.317993+0000", 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0397999999999999e-05 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.318017+0000", 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:2293 4.9 4:9441befd:::smithi20438004-8:head [write 1586353~512385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 57.789870722000003, 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066015969999999998, 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2293 2024-09-13T11:33:53.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271873+0000", 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.271874+0000", 2024-09-13T11:33:53.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.497e-06 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272359+0000", 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048419700000000001 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272399+0000", 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0754000000000003e-05 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272410+0000", 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272470+0000", 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0603999999999998e-05 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272575+0000", 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104901 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.272731+0000", 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015523300000000001 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.274392+0000", 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016616070000000001 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.276887+0000", 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024952360000000001 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278386+0000", 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014986540000000001 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278439+0000", 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2624999999999998e-05 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:55.278475+0000", 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5812999999999997e-05 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.118 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2494 4.d e63/31 4:b15c1843:::smithi20438004-33:head v 63'65, mlcod=63'62)", 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.412657013, 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064248049999999996, 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649087+0000", 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649088+0000", 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.78e-07 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649096+0000", 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3790000000000006e-06 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649108+0000", 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1751e-05 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649116+0000", 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3850000000000002e-06 2024-09-13T11:33:53.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649307+0000", 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019067199999999999 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.649325+0000", 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7859e-05 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655462+0000", 2024-09-13T11:33:53.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061369980000000003 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:00.655511+0000", 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9883000000000003e-05 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3068 3.1 e72/51 3:ba0ff2fc:::35d746116e015b742d2355e1d921a21f28a06eaf655b8cfcf0782553e2cf9a5a:head v 72'267, mlcod=70'265)", 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.445134150999998, 2024-09-13T11:33:53.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073641619999999996, 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620678+0000", 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040678880000000004 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620694+0000", 2024-09-13T11:33:53.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.611e-05 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620705+0000", 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1124999999999999e-05 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620768+0000", 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2626000000000007e-05 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620794+0000", 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6446000000000001e-05 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623920+0000", 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003126298 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623974+0000", 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3375000000000003e-05 2024-09-13T11:33:53.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3600 4.9 4:94e25a0d:::smithi20438004-3:head [write 527819~624406,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.910634683000001, 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076023949999999996, 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.128 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3600 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151112+0000", 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7070000000000001e-06 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151819+0000", 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00070706700000000003 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151834+0000", 2024-09-13T11:33:53.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4944e-05 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151841+0000", 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1760000000000004e-06 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151897+0000", 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5948e-05 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151966+0000", 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9283999999999996e-05 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.152076+0000", 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109633 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154405+0000", 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023291950000000001 2024-09-13T11:33:53.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154749+0000", 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034383100000000001 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158638+0000", 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038894250000000002 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158677+0000", 2024-09-13T11:33:53.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9167e-05 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158711+0000", 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4017999999999999e-05 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3987 4.b e81/52 4:da92e835:::smithi20438004-50:head v 81'69, mlcod=76'67)", 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:53.134 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.852279429999999, 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068559900000000002, 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7100000000000001e-07 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210320+0000", 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085537199999999997 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210332+0000", 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.224e-05 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210343+0000", 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0549e-05 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210424+0000", 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1209000000000001e-05 2024-09-13T11:33:53.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210450+0000", 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6673e-05 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216271+0000", 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058204060000000002 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216320+0000", 2024-09-13T11:33:53.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9370000000000003e-05 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4013 4.9 4:9441befd:::smithi20438004-8:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e81)", 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.367696895999998, 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017394468999999999, 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.139 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4013 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:33:53.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694048+0000", 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.026e-06 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694055+0000", 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9249999999999998e-06 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694065+0000", 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9049999999999992e-06 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694070+0000", 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4419999999999997e-06 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694114+0000", 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4341000000000001e-05 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694170+0000", 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5618999999999997e-05 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.711441+0000", 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017271211000000002 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4055 4.b e82/52 4:d738a69d:::smithi20438004-16:head v 82'73, mlcod=81'71)", 2024-09-13T11:33:53.143 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.457378902999999, 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066268300000000002, 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.604365+0000", 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8400000000000001e-07 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605461+0000", 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001096461 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605475+0000", 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3439e-05 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605482+0000", 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3000000000000004e-06 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605560+0000", 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8305000000000003e-05 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605584+0000", 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3723e-05 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.610952+0000", 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053675449999999996 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.610992+0000", 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9873000000000003e-05 2024-09-13T11:33:53.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4060 4.15 4:a9b0c9bd:::smithi20438004-37:head [write 1366083~511456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.446958026000001, 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073320010000000003, 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.148 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4060 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:33:53.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614787+0000", 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9500000000000001e-07 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615260+0000", 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047298899999999999 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615270+0000", 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615276+0000", 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.524e-06 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615325+0000", 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9530000000000002e-05 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615404+0000", 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9208000000000005e-05 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615492+0000", 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7163999999999994e-05 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.617037+0000", 2024-09-13T11:33:53.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001545374 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.619988+0000", 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029509470000000002 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622041+0000", 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002052824 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622091+0000", 2024-09-13T11:33:53.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622118+0000", 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7016e-05 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4061 4.15 4:a9b0c9bd:::smithi20438004-37:head [setxattr _header (56) in=7b,truncate 2016906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.446455064999999, 2024-09-13T11:33:53.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069574540000000001, 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4061 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0200000000000003e-07 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615291+0000", 2024-09-13T11:33:53.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615296+0000", 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9790000000000001e-06 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615299+0000", 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2040000000000002e-06 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616228+0000", 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000928213 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616282+0000", 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4147999999999997e-05 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616369+0000", 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7595000000000004e-05 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.617083+0000", 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071378200000000004 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622163+0000", 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050794780000000001 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622189+0000", 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6539000000000001e-05 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622216+0000", 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6443000000000001e-05 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622246+0000", 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0440999999999999e-05 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.160 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4062 4.15 4:a9b0c9bd:::smithi20438004-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.446421311999998, 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070487859999999996, 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4062 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615323+0000", 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615326+0000", 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615334+0000", 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9300000000000003e-06 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615342+0000", 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4610000000000002e-06 2024-09-13T11:33:53.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616576+0000", 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001234787 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616622+0000", 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549e-05 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622271+0000", 2024-09-13T11:33:53.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056493400000000001 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622313+0000", 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1616999999999998e-05 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622371+0000", 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8252999999999999e-05 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4414 4.b e86/52 4:da92e835:::smithi20438004-50:head v 86'79, mlcod=82'75)", 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:44.889871+0000", 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.1718724559999991, 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006469486, 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.889871+0000", 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.889871+0000", 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.889872+0000", 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2799999999999995e-07 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.889879+0000", 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9979999999999997e-06 2024-09-13T11:33:53.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.889889+0000", 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0030000000000001e-05 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.889897+0000", 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8590000000000005e-06 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.890592+0000", 2024-09-13T11:33:53.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069502799999999999 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.890607+0000", 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5142e-05 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.896300+0000", 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00569287 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:44.896341+0000", 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0731e-05 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4498 3.1 e91/51 3:801a20a0:::64b0ec958f46e0bde068907626863799a07e7e5b874bd667875147d240d0f7ef:head v 91'344, mlcod=85'342)", 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:49.517865+0000", 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5438789850000001, 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066688629999999997, 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.517865+0000", 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.517865+0000", 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.517866+0000", 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.415e-06 2024-09-13T11:33:53.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.521310+0000", 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034438160000000002 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.521326+0000", 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6132000000000001e-05 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.521337+0000", 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.064e-05 2024-09-13T11:33:53.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.521399+0000", 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1956000000000001e-05 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.521416+0000", 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7108e-05 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.524466+0000", 2024-09-13T11:33:53.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030501080000000002 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.524534+0000", 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7688000000000007e-05 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.174 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:53.176 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2168260 2024-09-13T11:33:53.176 INFO:tasks.rados.rados.0.smithi204.stdout:1893: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:53.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:33:53.177 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:53.177 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:53.177 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:53.178 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:53.179 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:53.179 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:53.179 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:53.179 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:53.180 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:53.180 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:53.180 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:53.173+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:53.183 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:53.183 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:53.184 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:53.201 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:53.219 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:53.219 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:53.236 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:53.236 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:53.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:53.276 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:53.275+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:53.291 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:53.291 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:53.291 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:53.291 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:53.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:33:53.337 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 656737 2024-09-13T11:33:53.377 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:53.376+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:53.406 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:53.406 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:53.406 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:53.407 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:53.407 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:53.411 INFO:tasks.rados.rados.0.smithi204.stdout:1890: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-13T11:33:53.411 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:53.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:33:53.478 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:53.476+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:53.523 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2279154 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:53.524 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.999233765, 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:33:53.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.531 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.310295624000005, 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.532 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:33:53.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:33:53.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.301953425999997, 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:33:53.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:33:53.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.281595422999999, 2024-09-13T11:33:53.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:33:53.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:33:53.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.550 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.273794663999993, 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:33:53.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.556 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 85.724287856000004, 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.557 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:33:53.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:33:53.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 82.762099320999994, 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.563 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:33:53.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:33:53.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:33:53.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1485 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~3966419,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 76.411632225000005, 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01527358, 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.570 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1485 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2300000000000002e-07 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110269+0000", 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110274+0000", 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2349999999999999e-06 2024-09-13T11:33:53.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110278+0000", 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6100000000000002e-06 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117718+0000", 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074403120000000001 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117821+0000", 2024-09-13T11:33:53.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103501 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.125539+0000", 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077171590000000003 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1636 4.8 4:1d367c64:::smithi20438004-45:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 72.521465058000004, 2024-09-13T11:33:53.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091050230000000003, 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1636 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000433+0000", 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1200000000000003e-07 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000436+0000", 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000443+0000", 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8619999999999997e-06 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000447+0000", 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8800000000000001e-06 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.001976+0000", 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001529774 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.002101+0000", 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124824 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009537+0000", 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074360850000000003 2024-09-13T11:33:53.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1637 4.8 4:1d367c64:::smithi20438004-45:head [sparse-read 0~2368645] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 72.521442528999998, 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010163363, 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.579 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1637 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000457+0000", 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000460+0000", 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3720000000000001e-06 2024-09-13T11:33:53.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000463+0000", 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.225e-06 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009568+0000", 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091047630000000001 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009626+0000", 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8142e-05 2024-09-13T11:33:53.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.010618+0000", 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099181000000000009 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1749 4.8 4:1d367c64:::smithi20438004-45:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.734196362999995, 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49694346099999998, 2024-09-13T11:33:53.583 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1749 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787703+0000", 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6840000000000001e-06 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787708+0000", 2024-09-13T11:33:53.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787721+0000", 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2405000000000001e-05 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787728+0000", 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.391e-06 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787780+0000", 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1897000000000001e-05 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787845+0000", 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5184000000000005e-05 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284644+0000", 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49679929099999998 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3085 4.d 4:b15c1843:::smithi20438004-33:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e72)", 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.247320942999998, 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012088290999999999, 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.588 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3085 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:33:53.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274578+0000", 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0609999999999998e-06 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274588+0000", 2024-09-13T11:33:53.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9119999999999993e-06 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274602+0000", 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3655000000000001e-05 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274613+0000", 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1199000000000001e-05 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.275921+0000", 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013075000000000001 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.275933+0000", 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1905999999999999e-05 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280509+0000", 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045760560000000002 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280683+0000", 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017461199999999999 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280858+0000", 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017475999999999999 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.284147+0000", 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032893900000000001 2024-09-13T11:33:53.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.284277+0000", 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012992 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286340+0000", 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020622779999999999 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286614+0000", 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274486 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286665+0000", 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0556000000000002e-05 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3500 4.d 4:bca378c8:::smithi20438004-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:33:53.595 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.762960519, 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026796718000000001, 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3500 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758939+0000", 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758952+0000", 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2337000000000001e-05 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758985+0000", 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3099000000000003e-05 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759005+0000", 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0197e-05 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759114+0000", 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010864799999999999 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759208+0000", 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4728999999999994e-05 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785733+0000", 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026525014999999999 2024-09-13T11:33:53.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3505 4.14 4:2c957d44:::smithi20438004-22:head [write 1940712~656106] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.746565725, 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010057587999999999, 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.600 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3505 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775333+0000", 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.488e-06 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776349+0000", 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010159909999999999 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776360+0000", 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1481000000000001e-05 2024-09-13T11:33:53.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776369+0000", 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776417+0000", 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8182e-05 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776474+0000", 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6669000000000002e-05 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776581+0000", 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010673199999999999 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.778281+0000", 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001700811 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785203+0000", 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069212960000000004 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785249+0000", 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6448e-05 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785290+0000", 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1340000000000001e-05 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785389+0000", 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8607e-05 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3568 4.d 4:b15c1843:::smithi20438004-33:head [write 477398~798659,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.236755240000001, 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010217238999999999, 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.606 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3568 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285148+0000", 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8449999999999999e-06 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286131+0000", 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00098294599999999995 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286146+0000", 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4712000000000001e-05 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286153+0000", 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6399999999999997e-06 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286206+0000", 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3284e-05 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286271+0000", 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286382+0000", 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011042799999999999 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.288353+0000", 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019708669999999998 2024-09-13T11:33:53.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.288371+0000", 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8538000000000001e-05 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295281+0000", 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069098720000000001 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295321+0000", 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0145000000000003e-05 2024-09-13T11:33:53.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295359+0000", 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8040000000000002e-05 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3632 4.8 4:1251e6cc:::smithi20438004-13:head [read 618102~2928175,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.290921126000001, 2024-09-13T11:33:53.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010153435000000001, 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3632 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230977+0000", 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9200000000000001e-07 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230984+0000", 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8990000000000004e-06 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230991+0000", 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5089999999999996e-06 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230998+0000", 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3999999999999997e-06 2024-09-13T11:33:53.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.235260+0000", 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042621330000000004 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.235319+0000", 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9549999999999997e-05 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.241130+0000", 2024-09-13T11:33:53.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058103520000000004 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3839 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.949279638, 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023869665000000002, 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:53.617 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3839 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572620+0000", 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0339999999999999e-06 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572630+0000", 2024-09-13T11:33:53.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0056e-05 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572658+0000", 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8756e-05 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572673+0000", 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572789+0000", 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011540900000000001 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572850+0000", 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1397999999999997e-05 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.596487+0000", 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023637162 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.621 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4056 4.b e82/52 4:d738a69d:::smithi20438004-16:head v 82'74, mlcod=81'71)", 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.605551+0000", 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.916346028, 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090727229999999996, 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605551+0000", 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605551+0000", 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605552+0000", 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3099999999999997e-07 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606348+0000", 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079557499999999997 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606356+0000", 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8659999999999995e-06 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606369+0000", 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2302e-05 2024-09-13T11:33:53.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606446+0000", 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7441999999999995e-05 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606468+0000", 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1440000000000001e-05 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614588+0000", 2024-09-13T11:33:53.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081204710000000006 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614624+0000", 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5896000000000003e-05 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4084 4.d 4:bca378c8:::smithi20438004-20:head [write 433713~482982] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:36.199349+0000", 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.322548451999999, 2024-09-13T11:33:53.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097880520000000002, 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4084 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.199349+0000", 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.199349+0000", 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.199351+0000", 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9379999999999999e-06 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.201928+0000", 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025777249999999999 2024-09-13T11:33:53.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.201951+0000", 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2248000000000001e-05 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.201963+0000", 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2092e-05 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.202041+0000", 2024-09-13T11:33:53.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7782999999999997e-05 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.202127+0000", 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6254999999999993e-05 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.202243+0000", 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116477 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.204044+0000", 2024-09-13T11:33:53.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018004749999999999 2024-09-13T11:33:53.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.204444+0000", 2024-09-13T11:33:53.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040071200000000001 2024-09-13T11:33:53.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:53.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.208902+0000", 2024-09-13T11:33:53.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044577870000000004 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.209103+0000", 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000200865 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.209137+0000", 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3695000000000002e-05 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4497 4.1c 4.de2cd65c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:48.158495+0000", 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.3634026329999998, 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.32777995100000001, 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4497 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.634 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158495+0000", 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158495+0000", 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158496+0000", 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4900000000000004e-07 2024-09-13T11:33:53.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158512+0000", 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6209e-05 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158526+0000", 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3798000000000001e-05 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158533+0000", 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4229999999999999e-06 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158597+0000", 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4256000000000003e-05 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158605+0000", 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5839999999999997e-06 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.486261+0000", 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.32765663 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.486275+0000", 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3101999999999999e-05 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:53.638 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:53.639 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:53.640 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:33:53.640 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:53.641 INFO:tasks.rados.rados.0.smithi204.stdout:1891: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-13T11:33:53.641 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2258779 2024-09-13T11:33:53.641 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:53.638+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:53.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:33:53.644 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:53.720 INFO:tasks.rados.rados.0.smithi204.stdout:1892: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:53.721 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:53.741 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:53.739+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:53.790 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:53.790 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:53.790 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:33:53.790 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:53.794 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 413315 2024-09-13T11:33:53.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:33:53.829 INFO:tasks.rados.rados.0.smithi204.stdout:1881: done (13 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1882: done (12 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1883: done (11 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1884: done (10 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1885: done (9 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1886: done (8 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1887: done (7 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1888: done (6 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1889: done (5 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1890: done (4 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1891: done (3 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1892: done (2 left) 2024-09-13T11:33:53.830 INFO:tasks.rados.rados.0.smithi204.stdout:1893: done (1 left) 2024-09-13T11:33:53.831 INFO:tasks.rados.rados.0.smithi204.stdout:1894: done (0 left) 2024-09-13T11:33:53.831 INFO:tasks.rados.rados.0.smithi204.stdout:1895: write (excl) oid 43 current snap is 0 2024-09-13T11:33:53.831 INFO:tasks.rados.rados.0.smithi204.stdout:1895: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-13T11:33:53.834 INFO:tasks.rados.rados.0.smithi204.stdout:1895: writing smithi20438004-43 from 490870 to 1109669 tid 1 2024-09-13T11:33:53.842 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:53.840+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:53.842 INFO:tasks.rados.rados.0.smithi204.stdout:1895: writing smithi20438004-43 from 1592676 to 2366820 tid 2 2024-09-13T11:33:53.845 INFO:tasks.rados.rados.0.smithi204.stdout:1895: writing smithi20438004-43 from 2793074 to 2890870 tid 3 2024-09-13T11:33:53.845 INFO:tasks.rados.rados.0.smithi204.stdout:1896: set_chunk oid 25 target oid 2024-09-13T11:33:53.857 INFO:tasks.rados.rados.0.smithi204.stdout:1896: set_chunk oid 25 offset: 65024 length: 724992 target oid 3c3c6d0a97cb75f9b8d58833219d228f668e92acf92f59538fec4caa33465719 offset: 0 snap -1 2024-09-13T11:33:53.857 INFO:tasks.rados.rados.0.smithi204.stdout:1897: read oid 23 snap -1 2024-09-13T11:33:53.857 INFO:tasks.rados.rados.0.smithi204.stdout:1897: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-13T11:33:53.858 INFO:tasks.rados.rados.0.smithi204.stdout:1898: read oid 2 snap -1 2024-09-13T11:33:53.858 INFO:tasks.rados.rados.0.smithi204.stdout:1898: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-13T11:33:53.858 INFO:tasks.rados.rados.0.smithi204.stdout:1899: read oid 40 snap -1 2024-09-13T11:33:53.858 INFO:tasks.rados.rados.0.smithi204.stdout:1899: expect deleted 2024-09-13T11:33:53.858 INFO:tasks.rados.rados.0.smithi204.stdout:1900: set_chunk oid 8 target oid 2024-09-13T11:33:53.873 INFO:tasks.rados.rados.0.smithi204.stdout:1900: set_chunk oid 8 offset: 216576 length: 1572864 target oid 2110e2e8252a0814dbb8b5dd4188666c70d9b5334a358f5df49b03c33e6a73d6 offset: 0 snap -1 2024-09-13T11:33:53.873 INFO:tasks.rados.rados.0.smithi204.stdout:1901: write (excl) oid 36 current snap is 0 2024-09-13T11:33:53.873 INFO:tasks.rados.rados.0.smithi204.stdout:1895: finishing write tid 1 to smithi20438004-43 2024-09-13T11:33:53.874 INFO:tasks.rados.rados.0.smithi204.stdout:1901: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-13T11:33:53.878 INFO:tasks.rados.rados.0.smithi204.stdout:1901: writing smithi20438004-36 from 507677 to 918078 tid 1 2024-09-13T11:33:53.884 INFO:tasks.rados.rados.0.smithi204.stdout:1901: writing smithi20438004-36 from 1367511 to 1885313 tid 2 2024-09-13T11:33:53.890 INFO:tasks.rados.rados.0.smithi204.stdout:1901: writing smithi20438004-36 from 2420497 to 2907677 tid 3 2024-09-13T11:33:53.890 INFO:tasks.rados.rados.0.smithi204.stdout:1902: write (excl) oid 4 current snap is 0 2024-09-13T11:33:53.890 INFO:tasks.rados.rados.0.smithi204.stdout:1895: finishing write tid 2 to smithi20438004-43 2024-09-13T11:33:53.890 INFO:tasks.rados.rados.0.smithi204.stdout:1895: finishing write tid 3 to smithi20438004-43 2024-09-13T11:33:53.890 INFO:tasks.rados.rados.0.smithi204.stdout:1902: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-13T11:33:53.895 INFO:tasks.rados.rados.0.smithi204.stdout:1902: writing smithi20438004-4 from 524484 to 1010134 tid 1 2024-09-13T11:33:53.901 INFO:tasks.rados.rados.0.smithi204.stdout:1902: writing smithi20438004-4 from 1425993 to 2087453 tid 2 2024-09-13T11:33:53.906 INFO:tasks.rados.rados.0.smithi204.stdout:1902: writing smithi20438004-4 from 2847920 to 2924484 tid 3 2024-09-13T11:33:53.906 INFO:tasks.rados.rados.0.smithi204.stdout:1903: delete oid 1 current snap is 0 2024-09-13T11:33:53.906 INFO:tasks.rados.rados.0.smithi204.stdout:1895: finishing write tid 4 to smithi20438004-43 2024-09-13T11:33:53.906 INFO:tasks.rados.rados.0.smithi204.stdout:1895: finishing write tid 5 to smithi20438004-43 2024-09-13T11:33:53.906 INFO:tasks.rados.rados.0.smithi204.stdout:1895: oid 43 updating version 0 to 128 2024-09-13T11:33:53.906 INFO:tasks.rados.rados.0.smithi204.stdout:1895: oid 43 updating version 128 to 130 2024-09-13T11:33:53.906 INFO:tasks.rados.rados.0.smithi204.stdout:1895: oid 43 version 130 is already newer than 129 2024-09-13T11:33:53.907 INFO:tasks.rados.rados.0.smithi204.stdout:1895: oid 43 updating version 130 to 131 2024-09-13T11:33:53.907 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 131 (ObjNum 410 snap 0 seq_num 410) dirty exists 2024-09-13T11:33:53.907 INFO:tasks.rados.rados.0.smithi204.stdout:1895: left oid 43 (ObjNum 410 snap 0 seq_num 410) 2024-09-13T11:33:53.907 INFO:tasks.rados.rados.0.smithi204.stdout:1896: finishing set_chunk to oid 25 2024-09-13T11:33:53.907 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 130 (ObjNum 377 snap 0 seq_num 377) dirty exists 2024-09-13T11:33:53.907 INFO:tasks.rados.rados.0.smithi204.stdout:1897: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-13T11:33:53.942 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:33:53.941+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:33:53.948 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:53.948 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:33:53.948 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:33:53.949 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:33:53.949 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:53.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:33:53.967 INFO:tasks.rados.rados.0.smithi204.stdout:1898: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-13T11:33:53.993 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1764924 2024-09-13T11:33:54.043 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:54.042+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:54.066 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:33:54.066 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:33:54.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:33:54.066 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:33:54.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 68.656145402999996, 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028959070000000001, 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:54.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408204+0000", 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.408205+0000", 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409400+0000", 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011945110000000001 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409410+0000", 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0159999999999999e-05 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409416+0000", 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:33:54.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409467+0000", 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1270999999999998e-05 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.409489+0000", 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1682999999999999e-05 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411056+0000", 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015677569999999999 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.411100+0000", 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4036999999999999e-05 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2315 4.0 e58/56 4:001d90c2:::smithi20438004-28:head v 58'53, mlcod=57'51)", 2024-09-13T11:33:54.071 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 57.944687594000001, 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031817899999999999, 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:54.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119662+0000", 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.119664+0000", 2024-09-13T11:33:54.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8330000000000001e-06 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120926+0000", 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012617710000000001 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120940+0000", 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4544e-05 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.120952+0000", 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1676e-05 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121050+0000", 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8177999999999994e-05 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.121071+0000", 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1174000000000001e-05 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122794+0000", 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00172215 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:56.122844+0000", 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0464000000000003e-05 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.12 60/59 [PushOp(4:4b088bf8:::smithi20438004-24:head, version: 60'85, data_included: [540672~892928,1630208~155648], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:4b088bf8:::smithi20438004-24:head@60'85, size: 3058874, copy_subset: [(0, 3058874)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1785856, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.751160012, 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003437201, 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.077 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313190+0000", 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.313191+0000", 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.046e-06 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315519+0000", 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023280670000000001 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315551+0000", 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2440000000000001e-05 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315574+0000", 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2286e-05 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315668+0000", 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929000000000001e-05 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.315679+0000", 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1e-05 2024-09-13T11:33:54.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.316627+0000", 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094843300000000005 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2354 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'52, mlcod=57'50)", 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.743163862000003, 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058622930000000002, 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:54.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321186+0000", 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.321187+0000", 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2200000000000003e-07 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323519+0000", 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002331891 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323532+0000", 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3494e-05 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323542+0000", 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323612+0000", 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9809999999999997e-05 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.323638+0000", 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6526999999999999e-05 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.326990+0000", 2024-09-13T11:33:54.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033519270000000002 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327048+0000", 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8158999999999997e-05 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2356 4.10 e60/59 4:0c9026a3:::smithi20438004-36:head v 60'54, mlcod=57'50)", 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:54.086 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.740041822000002, 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030409059999999999, 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324308+0000", 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0100000000000001e-07 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324312+0000", 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8469999999999998e-06 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324318+0000", 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6119999999999998e-06 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.324323+0000", 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2499999999999997e-06 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325363+0000", 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001040116 2024-09-13T11:33:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.325383+0000", 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9644999999999999e-05 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327310+0000", 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001927205 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:57.327349+0000", 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8930000000000002e-05 2024-09-13T11:33:54.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2513 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'86, mlcod=0'0)", 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.998519834, 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034473500000000001, 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.065830+0000", 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8099999999999999e-07 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066800+0000", 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096952400000000004 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066816+0000", 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6010000000000001e-05 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066826+0000", 2024-09-13T11:33:54.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0679000000000001e-05 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.066970+0000", 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014400900000000001 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.067020+0000", 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9595000000000002e-05 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069220+0000", 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022001640000000001 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.069277+0000", 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7188000000000002e-05 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2515 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'88, mlcod=63'86)", 2024-09-13T11:33:54.095 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.985305226999998, 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007260702, 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.079045+0000", 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4300000000000002e-07 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080302+0000", 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001256889 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080314+0000", 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.134e-05 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080319+0000", 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6699999999999999e-06 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080371+0000", 2024-09-13T11:33:54.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2039000000000001e-05 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080397+0000", 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5996e-05 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086251+0000", 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058537789999999999 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086305+0000", 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4146e-05 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2516 4.12 e63/61 4:4d4c8512:::smithi20438004-29:head v 63'89, mlcod=63'86)", 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.983929046999997, 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062074870000000002, 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.100 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080421+0000", 2024-09-13T11:33:54.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7199999999999999e-07 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080426+0000", 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8749999999999999e-06 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080434+0000", 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2930000000000001e-06 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.080443+0000", 2024-09-13T11:33:54.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3699999999999995e-06 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081319+0000", 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087608100000000002 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.081346+0000", 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7282999999999999e-05 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086595+0000", 2024-09-13T11:33:54.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052492759999999998 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:01.086628+0000", 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2836999999999997e-05 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.104 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2718 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'58, mlcod=58'56)", 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.940530293999998, 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039829779999999999, 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.123820+0000", 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125730+0000", 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001910213 2024-09-13T11:33:54.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125744+0000", 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3326000000000001e-05 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125752+0000", 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8879999999999997e-06 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125811+0000", 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9582e-05 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.125836+0000", 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4482e-05 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127747+0000", 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019112599999999999 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.127803+0000", 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5633000000000001e-05 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:2719 4.0 e66/56 4:001d90c2:::smithi20438004-28:head v 66'59, mlcod=60'57)", 2024-09-13T11:33:54.109 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.932753034000001, 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032139159999999998, 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131597+0000", 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.131598+0000", 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7999999999999993e-07 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132663+0000", 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010650010000000001 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132690+0000", 2024-09-13T11:33:54.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6970000000000001e-05 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132703+0000", 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.36e-05 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132837+0000", 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013366699999999999 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.132900+0000", 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3331999999999996e-05 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134769+0000", 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018688520000000001 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:06.134811+0000", 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1514000000000003e-05 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.113 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3042 4.12 e71/61 4:4b088bf8:::smithi20438004-24:head v 71'99, mlcod=70'97)", 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.948086388, 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036713660000000001, 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116263+0000", 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.116264+0000", 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8300000000000001e-07 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117477+0000", 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012129280000000001 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117486+0000", 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7100000000000002e-06 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117496+0000", 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6940000000000007e-06 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117579+0000", 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3207999999999993e-05 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.117606+0000", 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6622999999999999e-05 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119900+0000", 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022940899999999999 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.119935+0000", 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4929999999999999e-05 2024-09-13T11:33:54.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3122 4.12 e73/61 4:4b088bf8:::smithi20438004-24:head v 73'102, mlcod=71'100)", 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.688403246999997, 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031004679999999999, 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.375947+0000", 2024-09-13T11:33:54.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.74e-07 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377229+0000", 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001282245 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377237+0000", 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6939999999999995e-06 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377242+0000", 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377289+0000", 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6981999999999998e-05 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.377315+0000", 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6604000000000001e-05 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.379013+0000", 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016974959999999999 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.379047+0000", 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4156000000000003e-05 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3248 4.0 e74/56 4:04aeeab9:::smithi20438004-5:head v 74'73, mlcod=74'71)", 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:54.123 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.66054578, 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031644770000000002, 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403804+0000", 2024-09-13T11:33:54.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.403806+0000", 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6500000000000001e-06 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405022+0000", 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012166569999999999 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405039+0000", 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6942000000000001e-05 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405050+0000", 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0227000000000001e-05 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405112+0000", 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2916999999999994e-05 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.405139+0000", 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6594e-05 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.406922+0000", 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017830540000000001 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.406969+0000", 2024-09-13T11:33:54.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6436000000000001e-05 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3286 4.0 e74/56 4:001d90c2:::smithi20438004-28:head v 74'75, mlcod=74'73)", 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.469261396, 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048258800000000003, 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595088+0000", 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.595090+0000", 2024-09-13T11:33:54.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5319999999999999e-06 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596361+0000", 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001270953 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596378+0000", 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7363999999999998e-05 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596386+0000", 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9640000000000003e-06 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596440+0000", 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3544000000000001e-05 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.596459+0000", 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8864000000000001e-05 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.599857+0000", 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033986609999999999 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:19.599914+0000", 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3642 4.12 e76/61 4:4d4c8512:::smithi20438004-29:head v 76'111, mlcod=76'109)", 2024-09-13T11:33:54.132 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.818737455000001, 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003165579, 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245612+0000", 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.245615+0000", 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8499999999999998e-06 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246791+0000", 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001175773 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246805+0000", 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.401e-05 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246813+0000", 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7100000000000007e-06 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246879+0000", 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5889999999999994e-05 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.246909+0000", 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9937000000000002e-05 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.248741+0000", 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018326709999999999 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.248778+0000", 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6738e-05 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.136 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3846 4.12 e80/61 4:4ba9f189:::smithi20438004-14:head v 80'119, mlcod=78'117)", 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:30.589682+0000", 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.474667545999999, 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003140809, 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.589682+0000", 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.589682+0000", 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.589683+0000", 2024-09-13T11:33:54.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0900000000000002e-07 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591068+0000", 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001384786 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591081+0000", 2024-09-13T11:33:54.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3309000000000001e-05 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591091+0000", 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.066e-05 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591165+0000", 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3811000000000002e-05 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.591186+0000", 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0718999999999999e-05 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.592771+0000", 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015846409999999999 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.592823+0000", 2024-09-13T11:33:54.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2473999999999999e-05 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3847 4.12 e80/61 4:4ba9f189:::smithi20438004-14:head v 80'120, mlcod=78'117)", 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:30.595982+0000", 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.468367870000002, 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029379480000000001, 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.595982+0000", 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.595982+0000", 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.595983+0000", 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.61e-07 2024-09-13T11:33:54.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597086+0000", 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011028769999999999 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597098+0000", 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2595999999999999e-05 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597106+0000", 2024-09-13T11:33:54.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7649999999999993e-06 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597185+0000", 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8542999999999997e-05 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.597214+0000", 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9068999999999999e-05 2024-09-13T11:33:54.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.598876+0000", 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016628859999999999 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.598920+0000", 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3451000000000001e-05 2024-09-13T11:33:54.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4432 4.12 e86/61 4:4ba9f189:::smithi20438004-14:head v 86'137, mlcod=85'135)", 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:45.281190+0000", 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7831596730000001, 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029074270000000002, 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.281190+0000", 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.281190+0000", 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.281192+0000", 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9769999999999999e-06 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.282318+0000", 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001125863 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.282332+0000", 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3628999999999999e-05 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.282338+0000", 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7730000000000002e-06 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.282427+0000", 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8386000000000004e-05 2024-09-13T11:33:54.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.282446+0000", 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9077e-05 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.284034+0000", 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001587628 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:45.284098+0000", 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4094e-05 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4510 4.20 e91/90 4:04aeeab9:::smithi20438004-5:head v 91'89, mlcod=0'0)", 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:49.551754+0000", 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5125961730000004, 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087956430000000006, 2024-09-13T11:33:54.152 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.551754+0000", 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.551754+0000", 2024-09-13T11:33:54.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.551755+0000", 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.471e-06 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.558600+0000", 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068447830000000001 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.558628+0000", 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7705999999999999e-05 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.558642+0000", 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4643e-05 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.558735+0000", 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2975999999999999e-05 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.558763+0000", 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7322999999999999e-05 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.560493+0000", 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017303399999999999 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.560549+0000", 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6400999999999998e-05 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4511 4.20 e91/90 4:04aeeab9:::smithi20438004-5:head v 91'90, mlcod=0'0)", 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:49.560131+0000", 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5042185630000002, 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031765640000000002, 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.560131+0000", 2024-09-13T11:33:54.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.560131+0000", 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.560132+0000", 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2799999999999995e-07 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:33:54.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.561449+0000", 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001316528 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.561460+0000", 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1637000000000001e-05 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.561467+0000", 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4439999999999996e-06 2024-09-13T11:33:54.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.561532+0000", 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5302000000000006e-05 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.561551+0000", 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9110000000000002e-05 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.563254+0000", 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017033 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:49.563308+0000", 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3415000000000003e-05 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:33:54.162 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:33:54.165 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:33:54.168 INFO:tasks.rados.rados.0.smithi204.stdout:1900: finishing set_chunk to oid 8 2024-09-13T11:33:54.168 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 216576 length: 1572864 2024-09-13T11:33:54.168 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 0 (ObjNum 383 snap 0 seq_num 383) dirty exists 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1895: done (8 left) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1896: done (7 left) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1897: done (6 left) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1898: done (5 left) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1899: done (4 left) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1900: done (3 left) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1903: done (2 left) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1904: read oid 9 snap -1 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1904: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1905: write oid 12 current snap is 0 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1901: finishing write tid 1 to smithi20438004-36 2024-09-13T11:33:54.169 INFO:tasks.rados.rados.0.smithi204.stdout:1905: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1905: writing smithi20438004-12 from 558098 to 1194246 tid 1 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1905: writing smithi20438004-12 from 1826604 to 2491733 tid 2 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1906: read oid 27 snap -1 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1906: expect (ObjNum 391 snap 0 seq_num 391) 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1907: read oid 47 snap -1 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1907: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1908: write (excl) oid 13 current snap is 0 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1908: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1908: writing smithi20438004-13 from 574905 to 1286302 tid 1 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1908: writing smithi20438004-13 from 1885086 to 2410226 tid 2 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1908: writing smithi20438004-13 from 2930189 to 2974905 tid 3 2024-09-13T11:33:54.170 INFO:tasks.rados.rados.0.smithi204.stdout:1909: read oid 34 snap -1 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1909: expect (ObjNum 392 snap 0 seq_num 392) 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1910: write (excl) oid 44 current snap is 0 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1901: finishing write tid 2 to smithi20438004-36 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1901: finishing write tid 3 to smithi20438004-36 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1910: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1910: writing smithi20438004-44 from 591712 to 1378358 tid 1 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1910: writing smithi20438004-44 from 1943568 to 2612366 tid 2 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1911: read oid 21 snap -1 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1911: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1912: set_chunk oid 31 target oid 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1912: set_chunk oid 31 offset: 481280 length: 739840 target oid 3e1bd1c695e15f59f36fcc793fc7acb01c4bf8a922d3e1c4428349a9d38367d6 offset: 0 snap -1 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1913: tier_promote oid 37 2024-09-13T11:33:54.171 INFO:tasks.rados.rados.0.smithi204.stdout:1901: finishing write tid 4 to smithi20438004-36 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1901: finishing write tid 5 to smithi20438004-36 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1901: oid 36 updating version 0 to 71 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1901: oid 36 version 71 is already newer than 69 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1901: oid 36 updating version 71 to 72 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1901: oid 36 version 72 is already newer than 70 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 72 (ObjNum 411 snap 0 seq_num 411) dirty exists 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1901: left oid 36 (ObjNum 411 snap 0 seq_num 411) 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1902: finishing write tid 1 to smithi20438004-4 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1902: finishing write tid 2 to smithi20438004-4 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1902: finishing write tid 3 to smithi20438004-4 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1902: finishing write tid 4 to smithi20438004-4 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1901: done (11 left) 2024-09-13T11:33:54.172 INFO:tasks.rados.rados.0.smithi204.stdout:1914: set_chunk oid 8 target oid 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1914: set_chunk oid 8 offset: 11776 length: 683008 target oid 2773a4abff21ef753f038a7ff28d6038b1fc2a8aa2c327e0ae138a8a6eb55bb6 offset: 0 snap -1 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1915: tier_promote oid 38 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1902: finishing write tid 5 to smithi20438004-4 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1902: oid 4 updating version 0 to 97 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1902: oid 4 updating version 97 to 99 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1902: oid 4 version 99 is already newer than 98 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1902: oid 4 version 99 is already newer than 96 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 99 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1902: left oid 4 (ObjNum 412 snap 0 seq_num 412) 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1902: done (12 left) 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1916: write oid 41 current snap is 0 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:1904: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-13T11:33:54.173 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:54.174 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:54.162+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:54.235 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:54.265 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:54.264+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:54.286 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:54.287 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:54.287 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:54.287 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:54.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:33:54.307 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:54.366 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:54.364+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:54.380 INFO:tasks.rados.rados.0.smithi204.stdout:1905: finishing write tid 1 to smithi20438004-12 2024-09-13T11:33:54.380 INFO:tasks.rados.rados.0.smithi204.stdout:1906: expect (ObjNum 391 snap 0 seq_num 391) 2024-09-13T11:33:54.399 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:54.399 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:54.399 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:54.399 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:54.399 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:54.407 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:33:54.467 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:54.465+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:54.504 INFO:tasks.rados.rados.0.smithi204.stdout:1905: finishing write tid 2 to smithi20438004-12 2024-09-13T11:33:54.504 INFO:tasks.rados.rados.0.smithi204.stdout:1905: finishing write tid 3 to smithi20438004-12 2024-09-13T11:33:54.504 INFO:tasks.rados.rados.0.smithi204.stdout:1916: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-13T11:33:54.509 INFO:tasks.rados.rados.0.smithi204.stdout:1916: writing smithi20438004-41 from 608519 to 1070414 tid 1 2024-09-13T11:33:54.515 INFO:tasks.rados.rados.0.smithi204.stdout:1916: writing smithi20438004-41 from 1602050 to 2130859 tid 2 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1916: writing smithi20438004-41 from 2585035 to 3008519 tid 3 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1904: done (12 left) 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1906: done (11 left) 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1917: read oid 15 snap -1 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1917: expect (ObjNum 373 snap 0 seq_num 373) 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1918: read oid 3 snap -1 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1918: expect deleted 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1919: read oid 28 snap -1 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1919: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1920: tier_promote oid 9 2024-09-13T11:33:54.521 INFO:tasks.rados.rados.0.smithi204.stdout:1905: finishing write tid 4 to smithi20438004-12 2024-09-13T11:33:54.522 INFO:tasks.rados.rados.0.smithi204.stdout:1905: oid 12 updating version 0 to 106 2024-09-13T11:33:54.522 INFO:tasks.rados.rados.0.smithi204.stdout:1905: oid 12 updating version 106 to 108 2024-09-13T11:33:54.522 INFO:tasks.rados.rados.0.smithi204.stdout:1905: oid 12 version 108 is already newer than 107 2024-09-13T11:33:54.522 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 12 v 108 (ObjNum 414 snap 0 seq_num 414) dirty exists 2024-09-13T11:33:54.522 INFO:tasks.rados.rados.0.smithi204.stdout:1905: left oid 12 (ObjNum 414 snap 0 seq_num 414) 2024-09-13T11:33:54.522 INFO:tasks.rados.rados.0.smithi204.stdout:1907: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-13T11:33:54.525 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 860194 2024-09-13T11:33:54.554 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:54.554 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.460799359000006, 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011006991000000001, 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.555 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091278+0000", 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.091279+0000", 2024-09-13T11:33:54.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.091e-06 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098180+0000", 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069015359999999998 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098194+0000", 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.323e-05 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098202+0000", 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2879999999999993e-06 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098267+0000", 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4733999999999994e-05 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.098296+0000", 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9640000000000001e-05 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102225+0000", 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039281940000000003 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.102285+0000", 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0278000000000001e-05 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.559 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:225 3.0 e38/30 3:10cf6cc6:::smithi20438004-26 low_tier:head v 38'6, mlcod=38'3)", 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.403993869999994, 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010863593, 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148083+0000", 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.148085+0000", 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2810000000000001e-06 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153254+0000", 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051698430000000004 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153266+0000", 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1973e-05 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153274+0000", 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0059999999999992e-06 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153331+0000", 2024-09-13T11:33:54.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6492000000000001e-05 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153396+0000", 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5083e-05 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158906+0000", 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055099570000000002 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.158947+0000", 2024-09-13T11:33:54.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0958000000000003e-05 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.398781323999998, 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096623879999999992, 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.153296+0000", 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157238+0000", 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039414139999999999 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157253+0000", 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5566e-05 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157260+0000", 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.477e-06 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157315+0000", 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4857e-05 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.157341+0000", 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:33:54.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162899+0000", 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055583480000000003 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162958+0000", 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9169000000000002e-05 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.389123239, 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011410943999999999, 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162954+0000", 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.162955+0000", 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0669999999999999e-06 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167312+0000", 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004357072 2024-09-13T11:33:54.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167333+0000", 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0842000000000001e-05 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167341+0000", 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4430000000000003e-06 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167406+0000", 2024-09-13T11:33:54.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4245000000000006e-05 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.167433+0000", 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7500000000000001e-05 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174280+0000", 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068473969999999999 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.174365+0000", 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4377999999999997e-05 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.376295475999996, 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099135219999999993, 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.573 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175782+0000", 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.175783+0000", 2024-09-13T11:33:54.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6400000000000001e-07 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179929+0000", 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041461989999999997 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179943+0000", 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4731e-05 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.179953+0000", 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0359999999999995e-06 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180049+0000", 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6909000000000003e-05 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.180073+0000", 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4077e-05 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185641+0000", 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055674330000000001 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.185695+0000", 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4273e-05 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.577 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:258 4.3 4:c717f8db:::smithi20438004-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.338528210000007, 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017162947000000001, 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 258 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213549+0000", 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213551+0000", 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0760000000000001e-06 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213566+0000", 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5441999999999999e-05 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213585+0000", 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8623000000000001e-05 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213600+0000", 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4783999999999999e-05 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213667+0000", 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6973999999999999e-05 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.213747+0000", 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9794000000000002e-05 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.228488+0000", 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014740964 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229259+0000", 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00077144599999999996 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229743+0000", 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00048437000000000003 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230652+0000", 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090904600000000005 2024-09-13T11:33:54.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230688+0000", 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5474e-05 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230712+0000", 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3954000000000001e-05 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.583 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:259 4.1 4:8533c921:::smithi20438004-10:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.334252608, 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014548484, 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 259 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217825+0000", 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217826+0000", 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6360000000000001e-06 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217833+0000", 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8700000000000003e-06 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217844+0000", 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0825000000000001e-05 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217851+0000", 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-09-13T11:33:54.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217896+0000", 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4808000000000002e-05 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.217954+0000", 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7856000000000003e-05 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.229813+0000", 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011859533 2024-09-13T11:33:54.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231353+0000", 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001539503 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232188+0000", 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083500600000000001 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232284+0000", 2024-09-13T11:33:54.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6645999999999996e-05 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232325+0000", 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0071999999999999e-05 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232373+0000", 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8498999999999998e-05 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:266 4.4 4:2451796c:::smithi20438004-17:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.329752811999995, 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099464859999999992, 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.590 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 266 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222324+0000", 2024-09-13T11:33:54.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222325+0000", 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1900000000000001e-07 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222330+0000", 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7199999999999997e-06 2024-09-13T11:33:54.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222341+0000", 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0528e-05 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222347+0000", 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6409999999999996e-06 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222406+0000", 2024-09-13T11:33:54.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8449000000000002e-05 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.222487+0000", 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1167000000000004e-05 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,5", 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230197+0000", 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077105120000000001 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.230937+0000", 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000740159 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.231296+0000", 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035862399999999999 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232186+0000", 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089002899999999997 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232234+0000", 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7414999999999999e-05 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.232271+0000", 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7323000000000002e-05 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.596 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:283 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.318624677000003, 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011861159, 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 283 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233452+0000", 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233453+0000", 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2699999999999999e-07 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233457+0000", 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1150000000000004e-06 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233465+0000", 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233471+0000", 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3450000000000002e-06 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233526+0000", 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4790000000000002e-05 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233602+0000", 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6253000000000003e-05 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.237634+0000", 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040318580000000001 2024-09-13T11:33:54.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238163+0000", 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052849899999999998 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242711+0000", 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045478530000000001 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245235+0000", 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025239160000000002 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245286+0000", 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1739e-05 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245314+0000", 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7280999999999999e-05 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:284 4.1 4:8b052c81:::smithi20438004-35:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.318274665000004, 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011653294, 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 284 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.603 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233802+0000", 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233803+0000", 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4600000000000003e-07 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233808+0000", 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9559999999999996e-06 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233817+0000", 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9870000000000005e-06 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233824+0000", 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8290000000000002e-06 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233874+0000", 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9877999999999999e-05 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.233943+0000", 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9402999999999999e-05 2024-09-13T11:33:54.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238024+0000", 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040810350000000002 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.238440+0000", 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041507799999999999 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.242785+0000", 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043457890000000001 2024-09-13T11:33:54.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245363+0000", 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025773359999999999 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245416+0000", 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3156999999999999e-05 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245456+0000", 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0000000000000003e-05 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:286 4.5 4:a9b0c9bd:::smithi20438004-37:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.317292903999999, 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013081485, 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.609 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 286 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234784+0000", 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234785+0000", 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234790+0000", 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.651e-06 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234800+0000", 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234807+0000", 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.861e-06 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234868+0000", 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0582999999999999e-05 2024-09-13T11:33:54.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.234967+0000", 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9376000000000005e-05 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 9,11", 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054743730000000003 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241205+0000", 2024-09-13T11:33:54.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076331700000000001 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.241351+0000", 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619399999999999 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247766+0000", 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064151290000000003 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247825+0000", 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8739999999999997e-05 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247866+0000", 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747000000000001e-05 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:290 4.1 4:8d6fdfdb:::smithi20438004-41:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:54.615 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.312738754999998, 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014414148999999999, 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 290 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239338+0000", 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239339+0000", 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239344+0000", 2024-09-13T11:33:54.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2290000000000002e-06 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239354+0000", 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3589999999999997e-06 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239361+0000", 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.092e-06 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239406+0000", 2024-09-13T11:33:54.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5126000000000001e-05 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239452+0000", 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6192000000000001e-05 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.248478+0000", 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090262209999999992 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.249309+0000", 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083057300000000003 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253400+0000", 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040908760000000002 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253662+0000", 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026229500000000001 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253715+0000", 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2836e-05 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.253753+0000", 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7526e-05 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:292 4.1 4:84fa1f64:::smithi20438004-43:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.621 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.312315541999993, 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010951834000000001, 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 292 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239762+0000", 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8800000000000002e-07 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239765+0000", 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0299999999999998e-06 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239773+0000", 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8229999999999995e-06 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239779+0000", 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9900000000000002e-06 2024-09-13T11:33:54.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239831+0000", 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2326e-05 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.239910+0000", 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8623999999999998e-05 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.245574+0000", 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005663927 2024-09-13T11:33:54.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.246099+0000", 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052478300000000002 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.247963+0000", 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001864166 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250625+0000", 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002662228 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250676+0000", 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0989000000000003e-05 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250713+0000", 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7360000000000001e-05 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.627 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:317 4.1 4:8c30e60e:::smithi20438004-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.301191637000002, 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013134860999999999, 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.628 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 317 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250886+0000", 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250892+0000", 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250900+0000", 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250908+0000", 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.337e-06 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.250967+0000", 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8201000000000001e-05 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.251045+0000", 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8299000000000004e-05 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.258769+0000", 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007724323 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259599+0000", 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00082958200000000004 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260992+0000", 2024-09-13T11:33:54.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00139291 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263918+0000", 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029263190000000001 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263971+0000", 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2670999999999998e-05 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264020+0000", 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:330 4.3 4:c7f26bfe:::smithi20438004-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.299968821999997, 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011943621, 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.634 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 330 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252108+0000", 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252109+0000", 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8100000000000001e-07 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252116+0000", 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9199999999999998e-06 2024-09-13T11:33:54.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252129+0000", 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2768e-05 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252138+0000", 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5939999999999994e-06 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252205+0000", 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7015999999999997e-05 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252280+0000", 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4930000000000003e-05 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 11,12", 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259484+0000", 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00720434 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259994+0000", 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051054200000000001 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.260333+0000", 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033855000000000002 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.263947+0000", 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00361452 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264015+0000", 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7155000000000003e-05 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264052+0000", 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7305000000000003e-05 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.640 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:333 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.299678626000002, 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011814712, 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 333 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252399+0000", 2024-09-13T11:33:54.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252406+0000", 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252416+0000", 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-05 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252423+0000", 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9820000000000002e-06 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252486+0000", 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2805999999999995e-05 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.252540+0000", 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3897000000000003e-05 2024-09-13T11:33:54.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259138+0000", 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065977429999999997 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.259785+0000", 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064727700000000003 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264056+0000", 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042706710000000002 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264152+0000", 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5877000000000004e-05 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264180+0000", 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8674999999999999e-05 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.264213+0000", 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3015000000000002e-05 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:433 4.1 4:8a7c1b76:::smithi20438004-34:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 87.275127779000002, 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0098551839999999995, 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:54.647 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 433 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276949+0000", 2024-09-13T11:33:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276950+0000", 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4499999999999995e-07 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276955+0000", 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1059999999999999e-06 2024-09-13T11:33:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276965+0000", 2024-09-13T11:33:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.276977+0000", 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1518000000000001e-05 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277035+0000", 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8133000000000001e-05 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.277125+0000", 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0148999999999994e-05 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,15", 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282235+0000", 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005109461 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.282725+0000", 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049045700000000002 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286634+0000", 2024-09-13T11:33:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039087929999999998 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286707+0000", 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2692000000000001e-05 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286772+0000", 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5828999999999996e-05 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.286805+0000", 2024-09-13T11:33:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2134000000000001e-05 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:791 4.5 4:a5f8f9e2:::smithi20438004-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 86.852244869000003, 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019917450999999999, 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 791 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699832+0000", 2024-09-13T11:33:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699834+0000", 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3030000000000001e-06 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699840+0000", 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5989999999999998e-06 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699852+0000", 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1841999999999999e-05 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699861+0000", 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5259999999999993e-06 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699949+0000", 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8083000000000004e-05 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-13T11:33:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.699953+0000", 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3869999999999997e-06 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701189+0000", 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012354740000000001 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.701250+0000", 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1733e-05 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.719750+0000", 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018499504 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1374 4.1 4:8d6fdfdb:::smithi20438004-41:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.472177740999996, 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011338608999999999, 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1374 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079899+0000", 2024-09-13T11:33:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079900+0000", 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.04e-07 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079901+0000", 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.26e-06 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079905+0000", 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2039999999999999e-06 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.079908+0000", 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8200000000000001e-06 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081697+0000", 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017893080000000001 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.081802+0000", 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105231 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.091238+0000", 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094355819999999997 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1454 4.1 4:84fa1f64:::smithi20438004-43:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:54.665 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 77.716130898000003, 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025598754000000001, 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1454 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:54.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835946+0000", 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835947+0000", 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8500000000000002e-07 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835948+0000", 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5629999999999999e-06 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835952+0000", 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4259999999999999e-06 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.835954+0000", 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8550000000000002e-06 2024-09-13T11:33:54.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836019+0000", 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4416000000000001e-05 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.836271+0000", 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025171299999999998 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.861545+0000", 2024-09-13T11:33:54.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025274496 2024-09-13T11:33:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:54.670 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:54.670 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1908: finishing write tid 1 to smithi20438004-13 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1908: finishing write tid 2 to smithi20438004-13 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1908: finishing write tid 3 to smithi20438004-13 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1905: done (14 left) 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1907: done (13 left) 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1921: read oid 19 snap -1 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1921: expect (ObjNum 408 snap 0 seq_num 408) 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1922: delete oid 24 current snap is 0 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1908: finishing write tid 4 to smithi20438004-13 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1908: finishing write tid 5 to smithi20438004-13 2024-09-13T11:33:54.672 INFO:tasks.rados.rados.0.smithi204.stdout:1908: oid 13 updating version 0 to 58 2024-09-13T11:33:54.673 INFO:tasks.rados.rados.0.smithi204.stdout:1908: oid 13 version 58 is already newer than 55 2024-09-13T11:33:54.673 INFO:tasks.rados.rados.0.smithi204.stdout:1908: oid 13 version 58 is already newer than 57 2024-09-13T11:33:54.673 INFO:tasks.rados.rados.0.smithi204.stdout:1908: oid 13 version 58 is already newer than 56 2024-09-13T11:33:54.673 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 13 v 58 (ObjNum 415 snap 0 seq_num 415) dirty exists 2024-09-13T11:33:54.673 INFO:tasks.rados.rados.0.smithi204.stdout:1908: left oid 13 (ObjNum 415 snap 0 seq_num 415) 2024-09-13T11:33:54.673 INFO:tasks.rados.rados.0.smithi204.stdout:1909: expect (ObjNum 392 snap 0 seq_num 392) 2024-09-13T11:33:54.673 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2236621 2024-09-13T11:33:54.673 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:33:54.674 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:54.670+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:54.735 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:54.772 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:54.770+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:54.799 INFO:tasks.rados.rados.0.smithi204.stdout:1910: finishing write tid 1 to smithi20438004-44 2024-09-13T11:33:54.799 INFO:tasks.rados.rados.0.smithi204.stdout:1910: finishing write tid 2 to smithi20438004-44 2024-09-13T11:33:54.799 INFO:tasks.rados.rados.0.smithi204.stdout:1910: finishing write tid 3 to smithi20438004-44 2024-09-13T11:33:54.799 INFO:tasks.rados.rados.0.smithi204.stdout:1910: finishing write tid 4 to smithi20438004-44 2024-09-13T11:33:54.799 INFO:tasks.rados.rados.0.smithi204.stdout:1910: oid 44 updating version 0 to 87 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1910: oid 44 updating version 87 to 88 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1910: oid 44 updating version 88 to 89 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 44 v 89 (ObjNum 416 snap 0 seq_num 416) dirty exists 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1910: left oid 44 (ObjNum 416 snap 0 seq_num 416) 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1908: done (14 left) 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1909: done (13 left) 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1910: done (12 left) 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1922: done (11 left) 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1923: read oid 43 snap -1 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1923: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1924: read oid 32 snap -1 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1924: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1925: write (excl) oid 35 current snap is 0 2024-09-13T11:33:54.800 INFO:tasks.rados.rados.0.smithi204.stdout:1911: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-13T11:33:54.801 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:54.815 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:54.815 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:54.815 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-13T11:33:54.815 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:54.824 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:33:54.825 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:54.848 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 494091 2024-09-13T11:33:54.869 INFO:tasks.rados.rados.0.smithi204.stdout:1912: finishing set_chunk to oid 31 2024-09-13T11:33:54.870 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 31 set_chunk returned error code -95 offset: 481280 length: 739840 2024-09-13T11:33:54.870 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 0 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-13T11:33:54.870 INFO:tasks.rados.rados.0.smithi204.stdout:1913: got (0) Success 2024-09-13T11:33:54.870 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 37 v 49 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-09-13T11:33:54.870 INFO:tasks.rados.rados.0.smithi204.stdout:1925: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-13T11:33:54.872 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:54.871+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:54.877 INFO:tasks.rados.rados.0.smithi204.stdout:1925: writing smithi20438004-35 from 642133 to 1370879 tid 1 2024-09-13T11:33:54.884 INFO:tasks.rados.rados.0.smithi204.stdout:1925: writing smithi20438004-35 from 2119014 to 2651492 tid 2 2024-09-13T11:33:54.884 INFO:tasks.rados.rados.0.smithi204.stdout:1911: done (13 left) 2024-09-13T11:33:54.884 INFO:tasks.rados.rados.0.smithi204.stdout:1912: done (12 left) 2024-09-13T11:33:54.884 INFO:tasks.rados.rados.0.smithi204.stdout:1913: done (11 left) 2024-09-13T11:33:54.884 INFO:tasks.rados.rados.0.smithi204.stdout:1926: read oid 14 snap -1 2024-09-13T11:33:54.884 INFO:tasks.rados.rados.0.smithi204.stdout:1926: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-13T11:33:54.885 INFO:tasks.rados.rados.0.smithi204.stdout:1927: read oid 31 snap -1 2024-09-13T11:33:54.885 INFO:tasks.rados.rados.0.smithi204.stdout:1927: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:54.885 INFO:tasks.rados.rados.0.smithi204.stdout:1928: set_chunk oid 27 target oid 2024-09-13T11:33:54.906 INFO:tasks.rados.rados.0.smithi204.stdout:1928: set_chunk oid 27 offset: 148480 length: 2139648 target oid ca3a8052980711c3e55e90b7969e3ec4f7f8bbda7072715a16a7ec7b689db68c offset: 0 snap -1 2024-09-13T11:33:55.840 INFO:tasks.rados.rados.0.smithi204.stdout:1929: write oid 25 current snap is 0 2024-09-13T11:33:55.840 INFO:tasks.rados.rados.0.smithi204.stdout:1929: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-13T11:33:55.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:33:55.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-13T11:33:55.843 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:55.844 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-13T11:33:55.844 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-13T11:33:55.844 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-13T11:33:55.844 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:55.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:33:55.845 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:55.841+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:55.851 INFO:tasks.rados.rados.0.smithi204.stdout:1929: writing smithi20438004-25 from 658940 to 1062935 tid 1 2024-09-13T11:33:55.851 INFO:tasks.rados.rados.0.smithi204.stdout:1929: writing smithi20438004-25 from 1777496 to 2453632 tid 2 2024-09-13T11:33:55.851 INFO:tasks.rados.rados.0.smithi204.stdout:1929: writing smithi20438004-25 from 2950951 to 3058940 tid 3 2024-09-13T11:33:55.851 INFO:tasks.rados.rados.0.smithi204.stdout:1930: write oid 45 current snap is 0 2024-09-13T11:33:55.851 INFO:tasks.rados.rados.0.smithi204.stdout:1930: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1930: writing smithi20438004-45 from 675747 to 1154991 tid 1 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1930: writing smithi20438004-45 from 1835978 to 2372125 tid 2 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1915: got (0) Success 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 142 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1914: finishing set_chunk to oid 8 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 57 (ObjNum 383 snap 0 seq_num 383) dirty exists 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1916: finishing write tid 1 to smithi20438004-41 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1916: finishing write tid 2 to smithi20438004-41 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1914: done (15 left) 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1915: done (14 left) 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1931: write oid 48 current snap is 0 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1931: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-13T11:33:55.852 INFO:tasks.rados.rados.0.smithi204.stdout:1931: writing smithi20438004-48 from 692554 to 1247047 tid 1 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1931: writing smithi20438004-48 from 1778107 to 2457912 tid 2 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1931: writing smithi20438004-48 from 2889444 to 3092554 tid 3 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1932: delete oid 20 current snap is 0 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1920: got (0) Success 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 9 v 69 (ObjNum 387 snap 0 seq_num 387) dirty exists 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1916: finishing write tid 3 to smithi20438004-41 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1916: finishing write tid 4 to smithi20438004-41 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1916: finishing write tid 5 to smithi20438004-41 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1916: oid 41 updating version 0 to 157 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1916: oid 41 version 157 is already newer than 155 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1916: oid 41 version 157 is already newer than 154 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:1916: oid 41 version 157 is already newer than 156 2024-09-13T11:33:55.853 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 157 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:1916: left oid 41 (ObjNum 417 snap 0 seq_num 417) 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:1919: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:1917: expect (ObjNum 373 snap 0 seq_num 373) 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1002625 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:1921: expect (ObjNum 408 snap 0 seq_num 408) 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 279256 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:1924: expect (ObjNum 294 snap 0 seq_num 294) 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:1923: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 891170 2024-09-13T11:33:55.854 INFO:tasks.rados.rados.0.smithi204.stdout:1925: finishing write tid 1 to smithi20438004-35 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1925: finishing write tid 2 to smithi20438004-35 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1925: finishing write tid 3 to smithi20438004-35 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1925: finishing write tid 4 to smithi20438004-35 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1925: oid 35 updating version 0 to 158 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1925: oid 35 updating version 158 to 160 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1925: oid 35 version 160 is already newer than 159 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 35 v 160 (ObjNum 419 snap 0 seq_num 419) dirty exists 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1925: left oid 35 (ObjNum 419 snap 0 seq_num 419) 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:1926: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-13T11:33:55.855 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1051489 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1927: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1928: finishing set_chunk to oid 27 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 27 v 134 (ObjNum 391 snap 0 seq_num 391) dirty exists 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: finishing write tid 1 to smithi20438004-25 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: finishing write tid 2 to smithi20438004-25 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: finishing write tid 3 to smithi20438004-25 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: finishing write tid 4 to smithi20438004-25 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: finishing write tid 5 to smithi20438004-25 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: oid 25 updating version 0 to 143 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: oid 25 updating version 143 to 144 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: oid 25 version 144 is already newer than 142 2024-09-13T11:33:55.856 INFO:tasks.rados.rados.0.smithi204.stdout:1929: oid 25 version 144 is already newer than 141 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 144 (ObjNum 420 snap 0 seq_num 420) dirty exists 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1929: left oid 25 (ObjNum 420 snap 0 seq_num 420) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1930: finishing write tid 1 to smithi20438004-45 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1916: done (15 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1917: done (14 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1918: done (13 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1919: done (12 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1920: done (11 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1921: done (10 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1923: done (9 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1924: done (8 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1925: done (7 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1926: done (6 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1927: done (5 left) 2024-09-13T11:33:55.857 INFO:tasks.rados.rados.0.smithi204.stdout:1928: done (4 left) 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1929: done (3 left) 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1932: done (2 left) 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1933: read oid 36 snap -1 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1933: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1934: tier_evict oid 19 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1934: tier_evict oid 19 snap -1 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1930: finishing write tid 2 to smithi20438004-45 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1930: finishing write tid 3 to smithi20438004-45 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1935: read oid 20 snap -1 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1935: expect deleted 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1936: delete oid 4 current snap is 0 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1930: finishing write tid 4 to smithi20438004-45 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1930: oid 45 updating version 0 to 93 2024-09-13T11:33:55.858 INFO:tasks.rados.rados.0.smithi204.stdout:1930: oid 45 version 93 is already newer than 92 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1930: oid 45 updating version 93 to 94 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 94 (ObjNum 421 snap 0 seq_num 421) dirty exists 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1930: left oid 45 (ObjNum 421 snap 0 seq_num 421) 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: finishing write tid 1 to smithi20438004-48 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: finishing write tid 2 to smithi20438004-48 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: finishing write tid 3 to smithi20438004-48 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: finishing write tid 4 to smithi20438004-48 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: finishing write tid 5 to smithi20438004-48 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: oid 48 updating version 0 to 115 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: oid 48 updating version 115 to 117 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: oid 48 updating version 117 to 118 2024-09-13T11:33:55.859 INFO:tasks.rados.rados.0.smithi204.stdout:1931: oid 48 version 118 is already newer than 116 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 118 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1931: left oid 48 (ObjNum 422 snap 0 seq_num 422) 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1934: got (22) Invalid argument 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1930: done (5 left) 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1931: done (4 left) 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1934: done (3 left) 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1935: done (2 left) 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1936: done (1 left) 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1937: write oid 26 current snap is 0 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1937: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1937: writing smithi20438004-26 from 742975 to 1523215 tid 1 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1937: writing smithi20438004-26 from 1953553 to 2380685 tid 2 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1937: writing smithi20438004-26 from 2971713 to 3142975 tid 3 2024-09-13T11:33:55.860 INFO:tasks.rados.rados.0.smithi204.stdout:1938: tier_promote oid 43 2024-09-13T11:33:55.861 INFO:tasks.rados.rados.0.smithi204.stdout:1933: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-13T11:33:55.927 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1525187 2024-09-13T11:33:55.942 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:55.940+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:55.990 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:55.991 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:56.001 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4344.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 109.656211864, 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079674329999999995, 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4344", 2024-09-13T11:33:56.002 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343165+0000", 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343169+0000", 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8739999999999997e-06 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343177+0000", 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4819999999999994e-06 2024-09-13T11:33:56.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343189+0000", 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2306999999999999e-05 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343199+0000", 2024-09-13T11:33:56.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-06 2024-09-13T11:33:56.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343279+0000", 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0025999999999996e-05 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.343289+0000", 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0518e-05 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348428+0000", 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051380779999999999 2024-09-13T11:33:56.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348510+0000", 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2226999999999997e-05 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.348598+0000", 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8722999999999997e-05 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349014+0000", 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041537900000000001 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.349279+0000", 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026517700000000001 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351081+0000", 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001802292 2024-09-13T11:33:56.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351110+0000", 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8651000000000001e-05 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:06.351132+0000", 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2069000000000002e-05 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:21 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'4, mlcod=38'2)", 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 89.486228811999993, 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002707297, 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513148+0000", 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.513149+0000", 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.322e-06 2024-09-13T11:33:56.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514452+0000", 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001303153 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514468+0000", 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6045999999999998e-05 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514478+0000", 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4960000000000002e-06 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514531+0000", 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3442000000000001e-05 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.514546+0000", 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5211999999999999e-05 2024-09-13T11:33:56.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515821+0000", 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001274854 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:26.515855+0000", 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3772000000000001e-05 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:181 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'7, mlcod=38'5)", 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 88.978002313999994, 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002340637, 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021374+0000", 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.021376+0000", 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.013e-06 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022454+0000", 2024-09-13T11:33:56.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010781110000000001 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022469+0000", 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4199000000000001e-05 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022477+0000", 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2570000000000007e-06 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022532+0000", 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5636000000000001e-05 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.022548+0000", 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5769000000000001e-05 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023686+0000", 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011379929999999999 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.023715+0000", 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8659000000000001e-05 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:182 4.e e38/31 4:7984cf84:::smithi20438004-46:head v 38'8, mlcod=38'6)", 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 88.971936640999999, 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022898689999999999, 2024-09-13T11:33:56.020 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027440+0000", 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.027441+0000", 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7599999999999996e-07 2024-09-13T11:33:56.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028556+0000", 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011154439999999999 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028566+0000", 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8269999999999993e-06 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028571+0000", 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8550000000000001e-06 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028614+0000", 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2988999999999998e-05 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.028629+0000", 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5085999999999999e-05 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029703+0000", 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001073898 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.029730+0000", 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6993999999999999e-05 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.025 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:190 4.e e38/31 4:727deab8:::smithi20438004-48:head v 38'11, mlcod=38'9)", 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 88.942108911999995, 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022963139999999998, 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:56.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.057268+0000", 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058460+0000", 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192238 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058470+0000", 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9669999999999996e-06 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058475+0000", 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5399999999999997e-06 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058514+0000", 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8887999999999998e-05 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.058526+0000", 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1683e-05 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059537+0000", 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001011907 2024-09-13T11:33:56.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.059564+0000", 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6482000000000002e-05 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 88.604925605000005, 2024-09-13T11:33:56.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048595679999999999, 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394451+0000", 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.394452+0000", 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.09e-07 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397390+0000", 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002937977 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397401+0000", 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1172000000000001e-05 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397407+0000", 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8470000000000001e-06 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397452+0000", 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4705e-05 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.397467+0000", 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5622000000000001e-05 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399273+0000", 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001805823 2024-09-13T11:33:56.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:27.399310+0000", 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7512999999999998e-05 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1363 4.2 e43/31 4:4ba9f189:::smithi20438004-14:head v 43'50, mlcod=43'48)", 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 79.953793172000005, 2024-09-13T11:33:56.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021379849999999998, 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045583+0000", 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045585+0000", 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.604e-06 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045958+0000", 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000372672 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045978+0000", 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0262999999999999e-05 2024-09-13T11:33:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.045991+0000", 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2811e-05 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046079+0000", 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8667999999999999e-05 2024-09-13T11:33:56.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.046103+0000", 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4131e-05 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047664+0000", 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001560098 2024-09-13T11:33:56.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.047721+0000", 2024-09-13T11:33:56.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7738000000000002e-05 2024-09-13T11:33:56.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.042 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1410 4.4 e43/41 4:2451796c:::smithi20438004-17:head v 43'44, mlcod=43'42)", 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 79.870820664000007, 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022888489999999999, 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.043 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.044 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:56.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128556+0000", 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.128557+0000", 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3939999999999999e-06 2024-09-13T11:33:56.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129014+0000", 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045637100000000001 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129032+0000", 2024-09-13T11:33:56.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8685000000000002e-05 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129042+0000", 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129100+0000", 2024-09-13T11:33:56.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8238e-05 2024-09-13T11:33:56.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.129119+0000", 2024-09-13T11:33:56.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8258e-05 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130809+0000", 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001690045 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:36.130845+0000", 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5991000000000002e-05 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:33:56.050 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.968230758000004, 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071247150000000002, 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031146+0000", 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.031147+0000", 2024-09-13T11:33:56.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.579e-06 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036431+0000", 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00528394 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036449+0000", 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7329e-05 2024-09-13T11:33:56.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036457+0000", 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036529+0000", 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1506000000000003e-05 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.036551+0000", 2024-09-13T11:33:56.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2154000000000002e-05 2024-09-13T11:33:56.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038234+0000", 2024-09-13T11:33:56.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001683192 2024-09-13T11:33:56.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:37.038270+0000", 2024-09-13T11:33:56.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6408000000000001e-05 2024-09-13T11:33:56.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4631.0:1679 4.4 4.9c0290d4 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e46)", 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 73.940173053999999, 2024-09-13T11:33:56.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47635528100000002, 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4631", 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1679 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059203+0000", 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059205+0000", 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.812e-06 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059211+0000", 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1469999999999998e-06 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059222+0000", 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0560000000000001e-05 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059233+0000", 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0591000000000001e-05 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059286+0000", 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519e-05 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.059293+0000", 2024-09-13T11:33:56.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4690000000000002e-06 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535521+0000", 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.47622830100000002 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:42.535559+0000", 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7882000000000001e-05 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1701 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'54, mlcod=0'0)", 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 70.596925424000005, 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002436952, 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:56.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402451+0000", 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.402453+0000", 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.632e-06 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403268+0000", 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081553400000000003 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403289+0000", 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0353e-05 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403298+0000", 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4760000000000004e-06 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403369+0000", 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0858000000000005e-05 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.403397+0000", 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8518e-05 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404841+0000", 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001443228 2024-09-13T11:33:56.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.404888+0000", 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7352999999999999e-05 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1702 4.4 e50/49 4:2451796c:::smithi20438004-17:head v 50'55, mlcod=44'53)", 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 70.591399027999998, 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003287178, 2024-09-13T11:33:56.067 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407977+0000", 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.407979+0000", 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.046e-06 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409585+0000", 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016060479999999999 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409598+0000", 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3027000000000001e-05 2024-09-13T11:33:56.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409606+0000", 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2190000000000004e-06 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409676+0000", 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9791999999999999e-05 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.409707+0000", 2024-09-13T11:33:56.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1692000000000001e-05 2024-09-13T11:33:56.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411223+0000", 2024-09-13T11:33:56.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001515705 2024-09-13T11:33:56.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.411265+0000", 2024-09-13T11:33:56.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1649000000000001e-05 2024-09-13T11:33:56.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.072 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.072 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1705 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'62, mlcod=0'0)", 2024-09-13T11:33:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 70.583794628000007, 2024-09-13T11:33:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003138161, 2024-09-13T11:33:56.073 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415582+0000", 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.415583+0000", 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9699999999999995e-07 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416900+0000", 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001317251 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416926+0000", 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5622e-05 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.416942+0000", 2024-09-13T11:33:56.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6741000000000001e-05 2024-09-13T11:33:56.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417014+0000", 2024-09-13T11:33:56.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1298999999999996e-05 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.417042+0000", 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.851e-05 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418673+0000", 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016310949999999999 2024-09-13T11:33:56.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.418720+0000", 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6845999999999998e-05 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.079 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1706 4.2 e50/49 4:4d4c8512:::smithi20438004-29:head v 50'63, mlcod=45'61)", 2024-09-13T11:33:56.080 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:56.080 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 70.578462431000005, 2024-09-13T11:33:56.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003090597, 2024-09-13T11:33:56.080 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.080 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.080 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.080 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420914+0000", 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.420915+0000", 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-06 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422306+0000", 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001390753 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422317+0000", 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422322+0000", 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4360000000000001e-06 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422379+0000", 2024-09-13T11:33:56.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7003000000000003e-05 2024-09-13T11:33:56.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.422405+0000", 2024-09-13T11:33:56.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6424e-05 2024-09-13T11:33:56.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.423958+0000", 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015526680000000001 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:45.424005+0000", 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6619000000000002e-05 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.087 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1919 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'63, mlcod=53'61)", 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 66.033262453000006, 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002758908, 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.088 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966114+0000", 2024-09-13T11:33:56.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.966116+0000", 2024-09-13T11:33:56.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0370000000000001e-06 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967135+0000", 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010185260000000001 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967149+0000", 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3903999999999999e-05 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967157+0000", 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.405e-06 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967255+0000", 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8441000000000002e-05 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.967280+0000", 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5079000000000002e-05 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968828+0000", 2024-09-13T11:33:56.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001547936 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.968873+0000", 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4579999999999997e-05 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:1920 4.4 e53/49 4:2451796c:::smithi20438004-17:head v 53'64, mlcod=53'62)", 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 66.025311729999999, 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028733999999999999, 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974065+0000", 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.974066+0000", 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5700000000000001e-07 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975332+0000", 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012659349999999999 2024-09-13T11:33:56.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975345+0000", 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3402e-05 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975351+0000", 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4450000000000001e-06 2024-09-13T11:33:56.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975438+0000", 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6687999999999994e-05 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.975457+0000", 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8913e-05 2024-09-13T11:33:56.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976901+0000", 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014437180000000001 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:49.976938+0000", 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7441999999999998e-05 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.100 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2076 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'66, mlcod=53'64)", 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.080177965000004, 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022722770000000001, 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919199+0000", 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919200+0000", 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9000000000000005e-07 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919903+0000", 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070307200000000001 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919915+0000", 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2482e-05 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919922+0000", 2024-09-13T11:33:56.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5459999999999997e-06 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.919988+0000", 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6377999999999994e-05 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.920027+0000", 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8649999999999998e-05 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921419+0000", 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013924530000000001 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.921471+0000", 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1706000000000003e-05 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:2077 4.4 e55/49 4:2451796c:::smithi20438004-17:head v 55'67, mlcod=53'65)", 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:56.105 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.072035334000006, 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026288430000000001, 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927341+0000", 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.927342+0000", 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0200000000000001e-07 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928463+0000", 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011212640000000001 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928476+0000", 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3147e-05 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928484+0000", 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7319999999999998e-06 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928549+0000", 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5001999999999999e-05 2024-09-13T11:33:56.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.928565+0000", 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6124999999999999e-05 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929914+0000", 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001348859 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:32:51.929970+0000", 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5912000000000003e-05 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:3290 4.4 e74/56 4:2451796c:::smithi20438004-17:head v 74'77, mlcod=73'75)", 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.393030994999997, 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025948210000000002, 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.110 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606346+0000", 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.606347+0000", 2024-09-13T11:33:56.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607289+0000", 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094240700000000003 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607322+0000", 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3040999999999998e-05 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607350+0000", 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7331999999999999e-05 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607464+0000", 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114826 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.607491+0000", 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6271e-05 2024-09-13T11:33:56.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.608888+0000", 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013977309999999999 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.608940+0000", 2024-09-13T11:33:56.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1826000000000002e-05 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.4631.0:3291 4.4 e74/56 4:2451796c:::smithi20438004-17:head v 74'78, mlcod=73'76)", 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.387913697000002, 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022192150000000001, 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-13T11:33:56.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611463+0000", 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-13T11:33:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.611464+0000", 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.029e-06 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612224+0000", 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076036799999999998 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612238+0000", 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.342e-05 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612244+0000", 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4949999999999996e-06 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612295+0000", 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0392000000000001e-05 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.612314+0000", 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9568e-05 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.613631+0000", 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013171019999999999 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-13T11:33:19.613682+0000", 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0841000000000003e-05 2024-09-13T11:33:56.119 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.120 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.120 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.120 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-13T11:33:56.120 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-13T11:33:56.120 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-13T11:33:56.121 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:56.123 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:56.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:33:56.124 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:56.119+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:56.124 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:56.125 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:56.125 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:56.126 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:56.126 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:56.127 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:56.127 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:56.128 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:56.128 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:56.129 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:56.129 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: finishing write tid 1 to smithi20438004-26 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: finishing write tid 2 to smithi20438004-26 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: finishing write tid 3 to smithi20438004-26 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: finishing write tid 4 to smithi20438004-26 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: finishing write tid 5 to smithi20438004-26 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: oid 26 updating version 0 to 81 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: oid 26 updating version 81 to 83 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: oid 26 updating version 83 to 84 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: oid 26 version 84 is already newer than 82 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 84 (ObjNum 425 snap 0 seq_num 425) dirty exists 2024-09-13T11:33:56.132 INFO:tasks.rados.rados.0.smithi204.stdout:1937: left oid 26 (ObjNum 425 snap 0 seq_num 425) 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1938: got (0) Success 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 131 (ObjNum 410 snap 0 seq_num 410) dirty exists 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1933: done (2 left) 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1937: done (1 left) 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1938: done (0 left) 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1939: tier_flush oid 4 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1939: tier_flush oid 4 snap -1 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1940: write oid 8 current snap is 0 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1940: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1940: writing smithi20438004-8 from 759782 to 1331624 tid 1 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1940: writing smithi20438004-8 from 2012035 to 2699178 tid 2 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1941: write (excl) oid 38 current snap is 0 2024-09-13T11:33:56.133 INFO:tasks.rados.rados.0.smithi204.stdout:1939: got (2) No such file or directory 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1939: got expected ENOENT (src dne) 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1941: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1941: writing smithi20438004-38 from 776589 to 1423680 tid 1 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1941: writing smithi20438004-38 from 2070517 to 2501318 tid 2 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1941: writing smithi20438004-38 from 3026559 to 3176589 tid 3 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1939: done (2 left) 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1942: delete oid 1 current snap is 0 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1940: finishing write tid 1 to smithi20438004-8 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1940: finishing write tid 2 to smithi20438004-8 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1940: finishing write tid 3 to smithi20438004-8 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1940: finishing write tid 4 to smithi20438004-8 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1940: oid 8 updating version 0 to 60 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1940: oid 8 updating version 60 to 61 2024-09-13T11:33:56.134 INFO:tasks.rados.rados.0.smithi204.stdout:1940: oid 8 updating version 61 to 62 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 62 (ObjNum 426 snap 0 seq_num 426) dirty exists 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1940: left oid 8 (ObjNum 426 snap 0 seq_num 426) 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1941: finishing write tid 1 to smithi20438004-38 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1941: finishing write tid 2 to smithi20438004-38 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1940: done (2 left) 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1942: done (1 left) 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1943: write (excl) oid 50 current snap is 0 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1943: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1943: writing smithi20438004-50 from 410203 to 1207792 tid 1 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1943: writing smithi20438004-50 from 1787481 to 2221951 tid 2 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1943: writing smithi20438004-50 from 2681405 to 3210203 tid 3 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1944: read oid 47 snap -1 2024-09-13T11:33:56.135 INFO:tasks.rados.rados.0.smithi204.stdout:1944: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1945: read oid 11 snap -1 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1945: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1946: set_chunk oid 32 target oid 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1946: set_chunk oid 32 offset: 1222144 length: 36864 target oid 1c0273095382988333e2f2b5ae487cea460737ed9be65cbad9c5de537f95bf75 offset: 0 snap -1 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1947: set_chunk oid 42 target oid 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1947: set_chunk oid 42 offset: 991232 length: 1048064 target oid ad0b4dfa1f836fcb237b5cd826889057ebdfa1fde163eb52017956c4a589adef offset: 0 snap -1 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1948: write oid 21 current snap is 0 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1941: finishing write tid 3 to smithi20438004-38 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1948: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1948: writing smithi20438004-21 from 427010 to 899848 tid 1 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1948: writing smithi20438004-21 from 1329610 to 2024091 tid 2 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1948: writing smithi20438004-21 from 2708828 to 3227010 tid 3 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1949: read oid 39 snap -1 2024-09-13T11:33:56.136 INFO:tasks.rados.rados.0.smithi204.stdout:1949: expect (ObjNum 402 snap 0 seq_num 402) 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1950: write (excl) oid 17 current snap is 0 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1941: finishing write tid 4 to smithi20438004-38 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1941: finishing write tid 5 to smithi20438004-38 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1941: oid 38 updating version 0 to 146 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1941: oid 38 updating version 146 to 147 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1941: oid 38 updating version 147 to 148 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1941: oid 38 version 148 is already newer than 145 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 38 v 148 (ObjNum 427 snap 0 seq_num 427) dirty exists 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1941: left oid 38 (ObjNum 427 snap 0 seq_num 427) 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1943: finishing write tid 1 to smithi20438004-50 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1950: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1950: writing smithi20438004-17 from 443817 to 991904 tid 1 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1950: writing smithi20438004-17 from 1788092 to 2226231 tid 2 2024-09-13T11:33:56.137 INFO:tasks.rados.rados.0.smithi204.stdout:1950: writing smithi20438004-17 from 3019898 to 3243817 tid 3 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1941: done (8 left) 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1951: write (excl) oid 43 current snap is 0 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1951: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1951: writing smithi20438004-43 from 460624 to 1083960 tid 1 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1951: writing smithi20438004-43 from 1846574 to 2544724 tid 2 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1951: writing smithi20438004-43 from 3163674 to 3260624 tid 3 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1952: read oid 49 snap -1 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1952: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:56.138 INFO:tasks.rados.rados.0.smithi204.stdout:1953: set_chunk oid 2 target oid 2024-09-13T11:33:56.139 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:56.139 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1953: set_chunk oid 2 offset: 1117696 length: 271872 target oid ef0b388c32ad657fdde80865f14add19fe19009f948ae1e923ff63f4dc0f7a4c offset: 0 snap -1 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1954: read oid 24 snap -1 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1954: expect deleted 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1955: tier_evict oid 36 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1955: tier_evict oid 36 snap -1 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1943: finishing write tid 2 to smithi20438004-50 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1956: read oid 31 snap -1 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1956: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1957: tier_promote oid 10 2024-09-13T11:33:56.140 INFO:tasks.rados.rados.0.smithi204.stdout:1945: expect (ObjNum 282 snap 0 seq_num 282) 2024-09-13T11:33:56.155 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:56.155 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:56.177 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:56.177 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:56.178 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:56.189 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:56.189 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:56.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:33:56.188+0000 7fd86d23a640 0 -- 172.21.15.38:0/2936466988 >> v2:172.21.15.154:6804/538351706 conn(0x7fd8682159a0 msgr2=0x7fd868217d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:33:56.207 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:56.207 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:56.221 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:56.220+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:56.224 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:56.225 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:56.229 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:56.267 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:56.267 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:56.276 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:33:56.306 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 500274 2024-09-13T11:33:56.323 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:33:56.321+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:33:56.379 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:56.379 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:56.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:33:56.388 INFO:tasks.rados.rados.0.smithi204.stdout:1943: finishing write tid 3 to smithi20438004-50 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1946: finishing set_chunk to oid 32 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 1222144 length: 36864 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1943: finishing write tid 4 to smithi20438004-50 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1943: finishing write tid 5 to smithi20438004-50 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1943: oid 50 updating version 0 to 93 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1943: oid 50 version 93 is already newer than 92 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1943: oid 50 version 93 is already newer than 91 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1943: oid 50 version 93 is already newer than 90 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 50 v 93 (ObjNum 429 snap 0 seq_num 429) dirty exists 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1943: left oid 50 (ObjNum 429 snap 0 seq_num 429) 2024-09-13T11:33:56.389 INFO:tasks.rados.rados.0.smithi204.stdout:1944: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-13T11:33:56.414 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:56.424 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:56.423+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:56.452 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:56.490 INFO:tasks.rados.rados.0.smithi204.stdout:1947: finishing set_chunk to oid 42 2024-09-13T11:33:56.490 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 109 (ObjNum 396 snap 0 seq_num 396) dirty exists 2024-09-13T11:33:56.490 INFO:tasks.rados.rados.0.smithi204.stdout:1948: finishing write tid 1 to smithi20438004-21 2024-09-13T11:33:56.491 INFO:tasks.rados.rados.0.smithi204.stdout:1948: finishing write tid 2 to smithi20438004-21 2024-09-13T11:33:56.491 INFO:tasks.rados.rados.0.smithi204.stdout:1943: done (14 left) 2024-09-13T11:33:56.491 INFO:tasks.rados.rados.0.smithi204.stdout:1944: done (13 left) 2024-09-13T11:33:56.491 INFO:tasks.rados.rados.0.smithi204.stdout:1945: done (12 left) 2024-09-13T11:33:56.491 INFO:tasks.rados.rados.0.smithi204.stdout:1946: done (11 left) 2024-09-13T11:33:56.491 INFO:tasks.rados.rados.0.smithi204.stdout:1947: done (10 left) 2024-09-13T11:33:56.491 INFO:tasks.rados.rados.0.smithi204.stdout:1958: set_chunk oid 30 target oid 2024-09-13T11:33:56.493 INFO:tasks.rados.rados.0.smithi204.stdout:1958: set_chunk oid 30 offset: 139776 length: 65024 target oid b9b38af4a59de1678408a91e962a8f999564633239eed77bb408eed1510bc0e4 offset: 0 snap -1 2024-09-13T11:33:56.493 INFO:tasks.rados.rados.0.smithi204.stdout:1959: write oid 28 current snap is 0 2024-09-13T11:33:56.493 INFO:tasks.rados.rados.0.smithi204.stdout:1959: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-13T11:33:56.499 INFO:tasks.rados.rados.0.smithi204.stdout:1959: writing smithi20438004-28 from 477431 to 1176016 tid 1 2024-09-13T11:33:56.505 INFO:tasks.rados.rados.0.smithi204.stdout:1959: writing smithi20438004-28 from 1905056 to 2346864 tid 2 2024-09-13T11:33:56.511 INFO:tasks.rados.rados.0.smithi204.stdout:1959: writing smithi20438004-28 from 3074744 to 3277431 tid 3 2024-09-13T11:33:56.511 INFO:tasks.rados.rados.0.smithi204.stdout:1960: delete oid 44 current snap is 0 2024-09-13T11:33:56.511 INFO:tasks.rados.rados.0.smithi204.stdout:1948: finishing write tid 3 to smithi20438004-21 2024-09-13T11:33:56.511 INFO:tasks.rados.rados.0.smithi204.stdout:1948: finishing write tid 4 to smithi20438004-21 2024-09-13T11:33:56.511 INFO:tasks.rados.rados.0.smithi204.stdout:1948: finishing write tid 5 to smithi20438004-21 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:1948: oid 21 updating version 0 to 50 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:1948: oid 21 version 50 is already newer than 47 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:1948: oid 21 version 50 is already newer than 48 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:1948: oid 21 version 50 is already newer than 49 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 50 (ObjNum 430 snap 0 seq_num 430) dirty exists 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:1948: left oid 21 (ObjNum 430 snap 0 seq_num 430) 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:1949: expect (ObjNum 402 snap 0 seq_num 402) 2024-09-13T11:33:56.512 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:56.524 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:56.523+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:56.531 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:56.531 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:56.535 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-09-13T11:33:56.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:33:56.625 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:56.624+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:56.642 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:56.642 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:56.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:33:56.659 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:33:56.659 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:33:56.659 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:33:56.659 INFO:tasks.thrashosds.thrasher:Adding osd 13 2024-09-13T11:33:56.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 13 2024-09-13T11:33:56.666 INFO:tasks.rados.rados.0.smithi204.stdout:1950: finishing write tid 1 to smithi20438004-17 2024-09-13T11:33:56.666 INFO:tasks.rados.rados.0.smithi204.stdout:1950: finishing write tid 2 to smithi20438004-17 2024-09-13T11:33:56.666 INFO:tasks.rados.rados.0.smithi204.stdout:1950: finishing write tid 3 to smithi20438004-17 2024-09-13T11:33:56.666 INFO:tasks.rados.rados.0.smithi204.stdout:1950: finishing write tid 4 to smithi20438004-17 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1950: finishing write tid 5 to smithi20438004-17 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1950: oid 17 updating version 0 to 83 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1950: oid 17 updating version 83 to 84 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1950: oid 17 updating version 84 to 85 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1950: oid 17 version 85 is already newer than 82 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 85 (ObjNum 431 snap 0 seq_num 431) dirty exists 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1950: left oid 17 (ObjNum 431 snap 0 seq_num 431) 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1951: finishing write tid 1 to smithi20438004-43 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1951: finishing write tid 2 to smithi20438004-43 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1951: finishing write tid 3 to smithi20438004-43 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1951: finishing write tid 4 to smithi20438004-43 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1951: finishing write tid 5 to smithi20438004-43 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1951: oid 43 updating version 0 to 134 2024-09-13T11:33:56.667 INFO:tasks.rados.rados.0.smithi204.stdout:1951: oid 43 updating version 134 to 137 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:1951: oid 43 version 137 is already newer than 135 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:1951: oid 43 version 137 is already newer than 136 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 43 v 137 (ObjNum 432 snap 0 seq_num 432) dirty exists 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:1951: left oid 43 (ObjNum 432 snap 0 seq_num 432) 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:1953: finishing set_chunk to oid 2 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 1117696 length: 271872 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 0 (ObjNum 366 snap 0 seq_num 366) dirty exists 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:1955: got (22) Invalid argument 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:1952: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:56.668 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 68914 2024-09-13T11:33:56.726 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:56.725+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:56.754 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:56.755 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:56.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:33:56.777 INFO:tasks.rados.rados.0.smithi204.stdout:1956: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:56.807 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:56.827 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:56.826+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:56.838 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:56.865 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:56.865 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:56.869 INFO:tasks.rados.rados.0.smithi204.stdout:1957: got (0) Success 2024-09-13T11:33:56.869 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 10 v 126 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-09-13T11:33:56.869 INFO:tasks.rados.rados.0.smithi204.stdout:1958: finishing set_chunk to oid 30 2024-09-13T11:33:56.869 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 30 v 95 (ObjNum 399 snap 0 seq_num 399) dirty exists 2024-09-13T11:33:56.869 INFO:tasks.rados.rados.0.smithi204.stdout:1959: finishing write tid 1 to smithi20438004-28 2024-09-13T11:33:56.869 INFO:tasks.rados.rados.0.smithi204.stdout:1959: finishing write tid 2 to smithi20438004-28 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: finishing write tid 3 to smithi20438004-28 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: finishing write tid 4 to smithi20438004-28 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: finishing write tid 5 to smithi20438004-28 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: oid 28 updating version 0 to 89 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: oid 28 updating version 89 to 90 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: oid 28 updating version 90 to 91 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: oid 28 updating version 91 to 92 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 28 v 92 (ObjNum 433 snap 0 seq_num 433) dirty exists 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1959: left oid 28 (ObjNum 433 snap 0 seq_num 433) 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1948: done (12 left) 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1949: done (11 left) 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1950: done (10 left) 2024-09-13T11:33:56.870 INFO:tasks.rados.rados.0.smithi204.stdout:1951: done (9 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1952: done (8 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1953: done (7 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1954: done (6 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1955: done (5 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1956: done (4 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1957: done (3 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1958: done (2 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1959: done (1 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1960: done (0 left) 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1961: write (excl) oid 7 current snap is 0 2024-09-13T11:33:56.871 INFO:tasks.rados.rados.0.smithi204.stdout:1961: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-13T11:33:56.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:33:56.874 INFO:tasks.rados.rados.0.smithi204.stdout:1961: writing smithi20438004-7 from 511045 to 1076481 tid 1 2024-09-13T11:33:56.879 INFO:tasks.rados.rados.0.smithi204.stdout:1961: writing smithi20438004-7 from 1622020 to 2067497 tid 2 2024-09-13T11:33:56.886 INFO:tasks.rados.rados.0.smithi204.stdout:1961: writing smithi20438004-7 from 2729590 to 3286445 tid 3 2024-09-13T11:33:56.887 INFO:tasks.rados.rados.0.smithi204.stdout:1962: set_chunk oid 32 target oid 2024-09-13T11:33:56.901 INFO:tasks.rados.rados.0.smithi204.stdout:1962: set_chunk oid 32 offset: 552448 length: 1251328 target oid 750c6f727c4a6c360f03d350ded1003d7ef070b33cf3e998953fcf28f1460263 offset: 0 snap -1 2024-09-13T11:33:56.901 INFO:tasks.rados.rados.0.smithi204.stdout:1963: tier_evict oid 48 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1963: tier_evict oid 48 snap -1 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: finishing write tid 1 to smithi20438004-7 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: finishing write tid 2 to smithi20438004-7 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: finishing write tid 3 to smithi20438004-7 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: finishing write tid 4 to smithi20438004-7 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: finishing write tid 5 to smithi20438004-7 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: oid 7 updating version 0 to 97 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: oid 7 version 97 is already newer than 95 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: oid 7 version 97 is already newer than 96 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: oid 7 updating version 97 to 98 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 98 (ObjNum 435 snap 0 seq_num 435) dirty exists 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: left oid 7 (ObjNum 435 snap 0 seq_num 435) 2024-09-13T11:33:56.902 INFO:tasks.rados.rados.0.smithi204.stdout:1961: done (2 left) 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1964: read oid 28 snap -1 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1964: expect (ObjNum 433 snap 0 seq_num 433) 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1965: read oid 33 snap -1 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1965: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1966: tier_promote oid 17 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1967: delete oid 40 current snap is 0 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1962: finishing set_chunk to oid 32 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 552448 length: 1251328 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 32 v 0 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1963: got (22) Invalid argument 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:1966: got (0) Success 2024-09-13T11:33:56.903 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 85 (ObjNum 431 snap 0 seq_num 431) dirty exists 2024-09-13T11:33:56.906 INFO:tasks.rados.rados.0.smithi204.stdout:1962: done (5 left) 2024-09-13T11:33:56.906 INFO:tasks.rados.rados.0.smithi204.stdout:1963: done (4 left) 2024-09-13T11:33:56.906 INFO:tasks.rados.rados.0.smithi204.stdout:1966: done (3 left) 2024-09-13T11:33:56.906 INFO:tasks.rados.rados.0.smithi204.stdout:1967: done (2 left) 2024-09-13T11:33:56.906 INFO:tasks.rados.rados.0.smithi204.stdout:1968: tier_flush oid 29 2024-09-13T11:33:56.906 INFO:tasks.rados.rados.0.smithi204.stdout:1968: tier_flush oid 29 snap -1 2024-09-13T11:33:56.906 INFO:tasks.rados.rados.0.smithi204.stdout:1969: set_chunk oid 41 target oid 2024-09-13T11:33:56.919 INFO:tasks.rados.rados.0.smithi204.stdout:1969: set_chunk oid 41 offset: 138240 length: 1982464 target oid 0537267df1e5a4832a33c499e9b14b7e042ea2fb053a01f58d4dd0ee1820ad24 offset: 0 snap -1 2024-09-13T11:33:56.919 INFO:tasks.rados.rados.0.smithi204.stdout:1970: write oid 24 current snap is 0 2024-09-13T11:33:56.919 INFO:tasks.rados.rados.0.smithi204.stdout:1970: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-13T11:33:56.928 INFO:tasks.rados.rados.0.smithi204.stdout:1970: writing smithi20438004-24 from 544659 to 1260593 tid 1 2024-09-13T11:33:56.928 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:56.926+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:56.933 INFO:tasks.rados.rados.0.smithi204.stdout:1970: writing smithi20438004-24 from 1738984 to 2188130 tid 2 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1970: writing smithi20438004-24 from 2784436 to 3344659 tid 3 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1971: read oid 10 snap -1 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1971: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1972: read oid 49 snap -1 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1972: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1973: tier_flush oid 42 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1973: tier_flush oid 42 snap -1 2024-09-13T11:33:56.940 INFO:tasks.rados.rados.0.smithi204.stdout:1964: expect (ObjNum 433 snap 0 seq_num 433) 2024-09-13T11:33:56.979 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:33:56.977+0000 7f8352d34640 -1 mon.a@0(leader).osd e96 definitely_dead 0 2024-09-13T11:33:56.999 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:56.999 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:33:57.029 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:57.028+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:57.080 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.111 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.111 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:33:57.130 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:57.130+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:57.146 INFO:tasks.rados.rados.0.smithi204.stdout:1965: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-13T11:33:57.221 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.222 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:33:57.231 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:57.230+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:57.232 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1969: finishing set_chunk to oid 41 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 41 v 157 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1968: got (0) Success 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 29 v 0 (ObjNum 322 snap 0 seq_num 322) dirty exists 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1970: finishing write tid 1 to smithi20438004-24 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1970: finishing write tid 2 to smithi20438004-24 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1964: done (7 left) 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1965: done (6 left) 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1968: done (5 left) 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1969: done (4 left) 2024-09-13T11:33:57.289 INFO:tasks.rados.rados.0.smithi204.stdout:1974: tier_promote oid 31 2024-09-13T11:33:57.290 INFO:tasks.rados.rados.0.smithi204.stdout:1971: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-13T11:33:57.293 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 853081 2024-09-13T11:33:57.332 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:57.331+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:57.335 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.335 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.343 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2186563 2024-09-13T11:33:57.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:33:57.377 INFO:tasks.rados.rados.0.smithi204.stdout:1970: finishing write tid 3 to smithi20438004-24 2024-09-13T11:33:57.377 INFO:tasks.rados.rados.0.smithi204.stdout:1970: finishing write tid 4 to smithi20438004-24 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:1970: finishing write tid 5 to smithi20438004-24 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:1970: oid 24 updating version 0 to 143 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:1970: oid 24 updating version 143 to 144 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:1970: oid 24 version 144 is already newer than 140 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:1970: oid 24 version 144 is already newer than 142 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 24 v 144 (ObjNum 437 snap 0 seq_num 437) dirty exists 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:1970: left oid 24 (ObjNum 437 snap 0 seq_num 437) 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:1972: expect (ObjNum 382 snap 0 seq_num 382) 2024-09-13T11:33:57.378 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.414 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.433 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:57.432+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:57.448 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.448 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.451 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:33:57.489 INFO:tasks.rados.rados.0.smithi204.stdout:1973: got (0) Success 2024-09-13T11:33:57.489 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 42 v 0 (ObjNum 396 snap 0 seq_num 396) dirty exists 2024-09-13T11:33:57.489 INFO:tasks.rados.rados.0.smithi204.stdout:1974: got (0) Success 2024-09-13T11:33:57.489 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 62 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-13T11:33:57.489 INFO:tasks.rados.rados.0.smithi204.stdout:1970: done (4 left) 2024-09-13T11:33:57.489 INFO:tasks.rados.rados.0.smithi204.stdout:1971: done (3 left) 2024-09-13T11:33:57.490 INFO:tasks.rados.rados.0.smithi204.stdout:1972: done (2 left) 2024-09-13T11:33:57.490 INFO:tasks.rados.rados.0.smithi204.stdout:1973: done (1 left) 2024-09-13T11:33:57.490 INFO:tasks.rados.rados.0.smithi204.stdout:1974: done (0 left) 2024-09-13T11:33:57.490 INFO:tasks.rados.rados.0.smithi204.stdout:1975: write oid 26 current snap is 0 2024-09-13T11:33:57.490 INFO:tasks.rados.rados.0.smithi204.stdout:1975: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-13T11:33:57.496 INFO:tasks.rados.rados.0.smithi204.stdout:1975: writing smithi20438004-26 from 561466 to 1352649 tid 1 2024-09-13T11:33:57.503 INFO:tasks.rados.rados.0.smithi204.stdout:1975: writing smithi20438004-26 from 2081113 to 2673917 tid 2 2024-09-13T11:33:57.508 INFO:tasks.rados.rados.0.smithi204.stdout:1975: writing smithi20438004-26 from 3095506 to 3361466 tid 3 2024-09-13T11:33:57.508 INFO:tasks.rados.rados.0.smithi204.stdout:1976: tier_flush oid 47 2024-09-13T11:33:57.509 INFO:tasks.rados.rados.0.smithi204.stdout:1976: tier_flush oid 47 snap -1 2024-09-13T11:33:57.509 INFO:tasks.rados.rados.0.smithi204.stdout:1975: finishing write tid 1 to smithi20438004-26 2024-09-13T11:33:57.509 INFO:tasks.rados.rados.0.smithi204.stdout:1977: read oid 7 snap -1 2024-09-13T11:33:57.509 INFO:tasks.rados.rados.0.smithi204.stdout:1977: expect (ObjNum 435 snap 0 seq_num 435) 2024-09-13T11:33:57.509 INFO:tasks.rados.rados.0.smithi204.stdout:1978: read oid 13 snap -1 2024-09-13T11:33:57.509 INFO:tasks.rados.rados.0.smithi204.stdout:1978: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-13T11:33:57.509 INFO:tasks.rados.rados.0.smithi204.stdout:1979: read oid 39 snap -1 2024-09-13T11:33:57.510 INFO:tasks.rados.rados.0.smithi204.stdout:1979: expect (ObjNum 402 snap 0 seq_num 402) 2024-09-13T11:33:57.510 INFO:tasks.rados.rados.0.smithi204.stdout:1980: read oid 48 snap -1 2024-09-13T11:33:57.510 INFO:tasks.rados.rados.0.smithi204.stdout:1980: expect (ObjNum 422 snap 0 seq_num 422) 2024-09-13T11:33:57.510 INFO:tasks.rados.rados.0.smithi204.stdout:1981: read oid 37 snap -1 2024-09-13T11:33:57.510 INFO:tasks.rados.rados.0.smithi204.stdout:1981: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:57.510 INFO:tasks.rados.rados.0.smithi204.stdout:1982: write oid 25 current snap is 0 2024-09-13T11:33:57.510 INFO:tasks.rados.rados.0.smithi204.stdout:1982: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-09-13T11:33:57.517 INFO:tasks.rados.rados.0.smithi204.stdout:1982: writing smithi20438004-25 from 578273 to 1044705 tid 1 2024-09-13T11:33:57.523 INFO:tasks.rados.rados.0.smithi204.stdout:1982: writing smithi20438004-25 from 1739595 to 2192410 tid 2 2024-09-13T11:33:57.530 INFO:tasks.rados.rados.0.smithi204.stdout:1982: writing smithi20438004-25 from 2722929 to 3378273 tid 3 2024-09-13T11:33:57.530 INFO:tasks.rados.rados.0.smithi204.stdout:1983: set_chunk oid 11 target oid 2024-09-13T11:33:57.534 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:57.533+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:57.537 INFO:tasks.rados.rados.0.smithi204.stdout:1983: set_chunk oid 11 offset: 373760 length: 1178624 target oid 73840d8bd590ba4f9afd5dd661dd8e81f6cfcf79dd705e27256c65d38af8966b offset: 0 snap -1 2024-09-13T11:33:57.537 INFO:tasks.rados.rados.0.smithi204.stdout:1984: set_chunk oid 49 target oid 2024-09-13T11:33:57.542 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.13. 2024-09-13T11:33:57.556 INFO:tasks.rados.rados.0.smithi204.stdout:1984: set_chunk oid 49 offset: 265728 length: 1798144 target oid 3d6d78a5169d527853f055443a9b936f4cd67845feefb62138fac5ccc34aeb79 offset: 0 snap -1 2024-09-13T11:33:57.556 INFO:tasks.rados.rados.0.smithi204.stdout:1985: read oid 28 snap -1 2024-09-13T11:33:57.556 INFO:tasks.rados.rados.0.smithi204.stdout:1985: expect (ObjNum 433 snap 0 seq_num 433) 2024-09-13T11:33:57.556 INFO:tasks.rados.rados.0.smithi204.stdout:1986: write oid 22 current snap is 0 2024-09-13T11:33:57.556 INFO:tasks.rados.rados.0.smithi204.stdout:1986: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-13T11:33:57.562 INFO:tasks.thrashosds.thrasher:Added osd 13 2024-09-13T11:33:57.562 INFO:tasks.rados.rados.0.smithi204.stdout:1986: writing smithi20438004-22 from 595080 to 1136761 tid 1 2024-09-13T11:33:57.569 INFO:tasks.rados.rados.0.smithi204.stdout:1986: writing smithi20438004-22 from 1798077 to 2394550 tid 2 2024-09-13T11:33:57.572 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.572 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.575 INFO:tasks.rados.rados.0.smithi204.stdout:1986: writing smithi20438004-22 from 3150352 to 3395080 tid 3 2024-09-13T11:33:57.575 INFO:tasks.rados.rados.0.smithi204.stdout:1987: write oid 8 current snap is 0 2024-09-13T11:33:57.575 INFO:tasks.rados.rados.0.smithi204.stdout:1987: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-13T11:33:57.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:33:57.582 INFO:tasks.rados.rados.0.smithi204.stdout:1987: writing smithi20438004-8 from 611887 to 1345170 tid 1 2024-09-13T11:33:57.588 INFO:tasks.rados.rados.0.smithi204.stdout:1987: writing smithi20438004-8 from 1972912 to 2429396 tid 2 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1987: writing smithi20438004-8 from 2894128 to 3411887 tid 3 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1988: read oid 21 snap -1 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1988: expect (ObjNum 430 snap 0 seq_num 430) 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1989: tier_flush oid 23 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1989: tier_flush oid 23 snap -1 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1975: finishing write tid 2 to smithi20438004-26 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1975: finishing write tid 3 to smithi20438004-26 2024-09-13T11:33:57.594 INFO:tasks.rados.rados.0.smithi204.stdout:1975: finishing write tid 4 to smithi20438004-26 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:1975: finishing write tid 5 to smithi20438004-26 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:1975: oid 26 updating version 0 to 88 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:1975: oid 26 version 88 is already newer than 86 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:1975: oid 26 version 88 is already newer than 87 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:1975: oid 26 version 88 is already newer than 85 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 26 v 88 (ObjNum 438 snap 0 seq_num 438) dirty exists 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:1975: left oid 26 (ObjNum 438 snap 0 seq_num 438) 2024-09-13T11:33:57.595 INFO:tasks.rados.rados.0.smithi204.stdout:1980: expect (ObjNum 422 snap 0 seq_num 422) 2024-09-13T11:33:57.630 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.635 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:57.634+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:57.674 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2065756 2024-09-13T11:33:57.685 INFO:tasks.rados.rados.0.smithi204.stdout:1977: expect (ObjNum 435 snap 0 seq_num 435) 2024-09-13T11:33:57.688 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.688 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:33:57.737 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:57.736+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:57.756 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.838 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:57.837+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:57.839 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.839 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:57.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:33:57.899 INFO:tasks.rados.rados.0.smithi204.stdout:1979: expect (ObjNum 402 snap 0 seq_num 402) 2024-09-13T11:33:57.899 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:57.940 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:57.938+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:57.992 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:57.992 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:58.000 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:33:58.034 INFO:tasks.rados.rados.0.smithi204.stdout:1982: finishing write tid 1 to smithi20438004-25 2024-09-13T11:33:58.034 INFO:tasks.rados.rados.0.smithi204.stdout:1978: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-13T11:33:58.040 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:33:58.039+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:33:58.106 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:58.106 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:58.115 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:33:58.141 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:33:58.140+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:33:58.200 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:58.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:33:58.241 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:58.240+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:58.259 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:58.259 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:58.267 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:33:58.290 INFO:tasks.rados.rados.0.smithi204.stdout:1981: expect (ObjNum 358 snap 0 seq_num 358) 2024-09-13T11:33:58.311 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 678429 2024-09-13T11:33:58.330 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 582141 2024-09-13T11:33:58.342 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:33:58.341+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:33:58.346 INFO:tasks.rados.rados.0.smithi204.stdout:1982: finishing write tid 2 to smithi20438004-25 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1976: got (0) Success 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 0 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: finishing write tid 3 to smithi20438004-25 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: finishing write tid 4 to smithi20438004-25 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: finishing write tid 5 to smithi20438004-25 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: oid 25 updating version 0 to 145 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: oid 25 updating version 145 to 148 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: oid 25 version 148 is already newer than 146 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: oid 25 version 148 is already newer than 147 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 148 (ObjNum 439 snap 0 seq_num 439) dirty exists 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1982: left oid 25 (ObjNum 439 snap 0 seq_num 439) 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:1983: finishing set_chunk to oid 11 2024-09-13T11:33:58.347 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 120 (ObjNum 282 snap 0 seq_num 282) dirty exists 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1975: done (14 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1976: done (13 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1977: done (12 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1978: done (11 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1979: done (10 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1980: done (9 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1981: done (8 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1982: done (7 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1983: done (6 left) 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1990: tier_promote oid 48 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:1984: finishing set_chunk to oid 49 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 265728 length: 1798144 2024-09-13T11:33:58.348 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 0 (ObjNum 382 snap 0 seq_num 382) dirty exists 2024-09-13T11:33:58.349 INFO:tasks.rados.rados.0.smithi204.stdout:1985: expect (ObjNum 433 snap 0 seq_num 433) 2024-09-13T11:33:58.349 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:58.369 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:33:58.369 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:33:58.387 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:33:58.388 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:33:58.388 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:33:58.394 INFO:teuthology.orchestra.run.smithi204.stdout:ERROR: (22) Invalid argument 2024-09-13T11:33:58.394 INFO:teuthology.orchestra.run.smithi204.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:33:58.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:33:58.404 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:33:58.405 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:33:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:33:58.421 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:33:58.438 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:33:58.439 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:33:58.443 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:58.442+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:58.454 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:33:58.455 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:33:58.471 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:33:58.471 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:33:58.487 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:33:58.487 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:33:58.504 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:33:58.505 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:33:58.527 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:33:58.528 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:33:58.544 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:33:58.542+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1986: finishing write tid 1 to smithi20438004-22 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1986: finishing write tid 2 to smithi20438004-22 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1986: finishing write tid 3 to smithi20438004-22 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1986: finishing write tid 4 to smithi20438004-22 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1984: done (6 left) 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1985: done (5 left) 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1991: write (excl) oid 47 current snap is 0 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1986: finishing write tid 5 to smithi20438004-22 2024-09-13T11:33:58.545 INFO:tasks.rados.rados.0.smithi204.stdout:1986: oid 22 updating version 0 to 123 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:1986: oid 22 updating version 123 to 124 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:1986: oid 22 version 124 is already newer than 122 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:1986: oid 22 version 124 is already newer than 121 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 124 (ObjNum 440 snap 0 seq_num 440) dirty exists 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:1986: left oid 22 (ObjNum 440 snap 0 seq_num 440) 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:1987: finishing write tid 1 to smithi20438004-8 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:1987: finishing write tid 2 to smithi20438004-8 2024-09-13T11:33:58.546 INFO:tasks.rados.rados.0.smithi204.stdout:1991: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-09-13T11:33:58.546 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:33:58.547 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:33:58.549 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:33:58.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:33:58.553 INFO:tasks.rados.rados.0.smithi204.stdout:1991: writing smithi20438004-47 from 628694 to 1037226 tid 1 2024-09-13T11:33:58.554 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:58.554 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:58.554 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:58.554 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:58.558 INFO:tasks.rados.rados.0.smithi204.stdout:1991: writing smithi20438004-47 from 1515041 to 2115183 tid 2 2024-09-13T11:33:58.564 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1991: writing smithi20438004-47 from 2805198 to 3357310 tid 3 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1986: done (5 left) 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1992: read oid 31 snap -1 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1992: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1993: read oid 7 snap -1 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1993: expect (ObjNum 435 snap 0 seq_num 435) 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1994: write (excl) oid 16 current snap is 0 2024-09-13T11:33:58.566 INFO:tasks.rados.rados.0.smithi204.stdout:1994: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-13T11:33:58.572 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:33:58.572 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:33:58.572 INFO:tasks.rados.rados.0.smithi204.stdout:1994: writing smithi20438004-16 from 645501 to 1129282 tid 1 2024-09-13T11:33:58.577 INFO:tasks.rados.rados.0.smithi204.stdout:1994: writing smithi20438004-16 from 1573523 to 2033676 tid 2 2024-09-13T11:33:58.585 INFO:tasks.rados.rados.0.smithi204.stdout:1994: writing smithi20438004-16 from 2548974 to 3312358 tid 3 2024-09-13T11:33:58.585 INFO:tasks.rados.rados.0.smithi204.stdout:1995: read oid 19 snap -1 2024-09-13T11:33:58.585 INFO:tasks.rados.rados.0.smithi204.stdout:1995: expect (ObjNum 408 snap 0 seq_num 408) 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1996: tier_promote oid 2 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: finishing write tid 3 to smithi20438004-8 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: finishing write tid 4 to smithi20438004-8 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: finishing write tid 5 to smithi20438004-8 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: oid 8 updating version 0 to 63 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: oid 8 updating version 63 to 64 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: oid 8 updating version 64 to 65 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: oid 8 updating version 65 to 66 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 66 (ObjNum 441 snap 0 seq_num 441) dirty exists 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: left oid 8 (ObjNum 441 snap 0 seq_num 441) 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1987: done (9 left) 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1997: write (excl) oid 14 current snap is 0 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:1988: expect (ObjNum 430 snap 0 seq_num 430) 2024-09-13T11:33:58.586 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:58.589 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:33:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:33:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-09-13T11:33:58.588+0000 7f89daffd640 0 -- 172.21.15.38:0/3026103749 >> v2:172.21.15.204:6812/501019151 conn(0x7f89dc21d4e0 msgr2=0x7f89dc21f8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-13T11:33:58.607 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:33:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:33:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:33:58.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:33:58.636 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 646196 2024-09-13T11:33:58.645 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:58.644+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:58.675 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:58.675 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:58.675 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:58.675 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:58.675 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:58.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:33:58.722 INFO:tasks.rados.rados.0.smithi204.stdout:1991: finishing write tid 1 to smithi20438004-47 2024-09-13T11:33:58.722 INFO:tasks.rados.rados.0.smithi204.stdout:1991: finishing write tid 2 to smithi20438004-47 2024-09-13T11:33:58.722 INFO:tasks.rados.rados.0.smithi204.stdout:1993: expect (ObjNum 435 snap 0 seq_num 435) 2024-09-13T11:33:58.759 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2192897 2024-09-13T11:33:58.760 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:33:58.758+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:33:58.828 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:58.828 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:58.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:58.828 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:58.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.828 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:58.828 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.728686693, 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:33:58.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:33:58.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:33:58.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:33:58.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:33:58.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:33:58.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:33:58.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.716252548, 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:33:58.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:33:58.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:33:58.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.706298954000005, 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.839 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.840 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.843 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.675581147000003, 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.656757827999996, 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:33:58.848 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:58.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.643238479000004, 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:33:58.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:33:58.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:33:58.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.633598145999997, 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:33:58.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:33:58.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:33:58.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.623292645999996, 2024-09-13T11:33:58.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:33:58.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:33:58.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:33:58.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 91.612906703999997, 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:33:58.867 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.870 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:36 3.3 3:e2816a99:::d4bb749789c90c884ffd20ac62f804c7a6b80502d14432fa627e1239249ee2da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.555379305000002, 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013305613000000001, 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270480+0000", 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2399999999999998e-07 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270655+0000", 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017481300000000001 2024-09-13T11:33:58.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270683+0000", 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8340999999999999e-05 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270693+0000", 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271029+0000", 2024-09-13T11:33:58.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033640200000000002 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271038+0000", 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278355+0000", 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073166510000000004 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278481+0000", 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125955 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278657+0000", 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176816 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280615+0000", 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019576810000000002 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283538+0000", 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029229030000000001 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283592+0000", 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4366000000000001e-05 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283671+0000", 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8337000000000006e-05 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283785+0000", 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114412 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:40 3.3 3:c7a12847:::768a08ae37e9bf19369a517906ebd209c1f2b8ad3684596d644da7d48b42114d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.555136199000003, 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013347114, 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.877 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270839+0000", 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115369 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270847+0000", 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270854+0000", 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271073+0000", 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021964400000000001 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271080+0000", 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7909999999999999e-06 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278928+0000", 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078479819999999999 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279058+0000", 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130102 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279300+0000", 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024166300000000001 2024-09-13T11:33:58.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280641+0000", 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013409229999999999 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283545+0000", 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002904401 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283927+0000", 2024-09-13T11:33:58.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038193199999999997 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283964+0000", 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7117999999999999e-05 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284070+0000", 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010545899999999999 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:41 3.3 3:e9fa9a15:::3c65c6c9770b7a29cd5a9b6dc5b0b39ab23cb2a64bfb19442efcb911d255e067:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.554992585999997, 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013499547000000001, 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270867+0000", 2024-09-13T11:33:58.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270893+0000", 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6057000000000001e-05 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270901+0000", 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270915+0000", 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4450000000000001e-05 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271104+0000", 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018904600000000001 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271111+0000", 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6039999999999998e-06 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280674+0000", 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095629140000000005 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280811+0000", 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136802 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281027+0000", 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000216617 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283802+0000", 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027746540000000001 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283884+0000", 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2167000000000001e-05 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284178+0000", 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029347899999999997 2024-09-13T11:33:58.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284246+0000", 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8609999999999995e-05 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284366+0000", 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011950799999999999 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:43 3.3 3:dc8816b1:::1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.554931890999995, 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013665096999999999, 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270933+0000", 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2410000000000003e-06 2024-09-13T11:33:58.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270939+0000", 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5810000000000002e-06 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270945+0000", 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271134+0000", 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000189161 2024-09-13T11:33:58.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271140+0000", 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2999999999999998e-06 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281407+0000", 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010266226 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281549+0000", 2024-09-13T11:33:58.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142583 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281737+0000", 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018791500000000001 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283810+0000", 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002072571 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284127+0000", 2024-09-13T11:33:58.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000317516 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284417+0000", 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029000899999999998 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284480+0000", 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3136e-05 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284592+0000", 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111712 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 62.981104019, 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43316489499999999, 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.897 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:33:58.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844756+0000", 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844760+0000", 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0550000000000001e-06 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844769+0000", 2024-09-13T11:33:58.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6129999999999993e-06 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844776+0000", 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0280000000000003e-06 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7639000000000002e-05 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844841+0000", 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1049999999999997e-06 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277896+0000", 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43305493499999997 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277920+0000", 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3614000000000001e-05 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.901 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 62.981052386000002, 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43315226600000001, 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:58.902 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844807+0000", 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3200000000000003e-07 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844811+0000", 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.799e-06 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844827+0000", 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5461999999999999e-05 2024-09-13T11:33:58.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.126e-06 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5369999999999999e-05 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844863+0000", 2024-09-13T11:33:58.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.658e-06 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277951+0000", 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308847900000003 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277959+0000", 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 62.981010675, 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43314228399999999, 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844849+0000", 2024-09-13T11:33:58.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0100000000000005e-07 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844853+0000", 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9589999999999997e-06 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4110000000000001e-06 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844864+0000", 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8930000000000004e-06 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844892+0000", 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8042e-05 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844898+0000", 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5049999999999998e-06 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277984+0000", 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308647700000003 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277990+0000", 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3960000000000003e-06 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 153896~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 61.543359762999998, 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014013433, 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282500+0000", 2024-09-13T11:33:58.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5499999999999998e-07 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282503+0000", 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.563e-06 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282510+0000", 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8820000000000003e-06 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282516+0000", 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282562+0000", 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5868999999999997e-05 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282570+0000", 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296085+0000", 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013515255 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296187+0000", 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101457 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296512+0000", 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588099999999998 2024-09-13T11:33:58.916 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 447518~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 61.543275270000002, 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014496826000000001, 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:58.917 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5000000000000002e-07 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282589+0000", 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4220000000000002e-06 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282595+0000", 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8789999999999997e-06 2024-09-13T11:33:58.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282606+0000", 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0229e-05 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282652+0000", 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6610999999999998e-05 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282661+0000", 2024-09-13T11:33:58.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2640000000000008e-06 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296550+0000", 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013888981999999999 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296628+0000", 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8564000000000002e-05 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.297080+0000", 2024-09-13T11:33:58.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000452225 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4497 4.1c 4:3a6b347b:::smithi20438004-49:head [read 39936~1781760] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:49.488729+0000", 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.3371302820000004, 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017108786000000001, 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:58.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4497 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488729+0000", 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488729+0000", 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488730+0000", 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9939999999999999e-06 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488736+0000", 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9859999999999999e-06 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488752+0000", 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5872999999999999e-05 2024-09-13T11:33:58.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488764+0000", 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1165000000000001e-05 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.491323+0000", 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025592029999999999 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.491338+0000", 2024-09-13T11:33:58.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4849e-05 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.499774+0000", 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084368629999999993 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.499959+0000", 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000184737 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.505837+0000", 2024-09-13T11:33:58.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058781160000000001 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4499 4.1c 4:3a6b347b:::smithi20438004-49:head [assert-version v44,set-chunk] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:49.540567+0000", 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 9.2852919160000003, 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015634990000000001, 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:58.927 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4499 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540567+0000", 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540567+0000", 2024-09-13T11:33:58.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540568+0000", 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2100000000000001e-06 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540575+0000", 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2559999999999996e-06 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540585+0000", 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2469999999999997e-06 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540590+0000", 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7250000000000002e-06 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.542963+0000", 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002372809 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.543053+0000", 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0259000000000004e-05 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4,6,10,15", 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.547232+0000", 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041787359999999997 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.547764+0000", 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053142700000000001 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.547973+0000", 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020973500000000001 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.549921+0000", 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00194754 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.555923+0000", 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060020309999999997 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.555976+0000", 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3100999999999999e-05 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.556049+0000", 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2733999999999999e-05 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.556202+0000", 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015317999999999999 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.933 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:58.934 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:58.934 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:58.935 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:33:58.936 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:33:58.934+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: finishing write tid 3 to smithi20438004-47 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: finishing write tid 4 to smithi20438004-47 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: finishing write tid 5 to smithi20438004-47 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: oid 47 updating version 0 to 49 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: oid 47 updating version 49 to 50 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: oid 47 updating version 50 to 51 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: oid 47 updating version 51 to 52 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 47 v 52 (ObjNum 442 snap 0 seq_num 442) dirty exists 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1991: left oid 47 (ObjNum 442 snap 0 seq_num 442) 2024-09-13T11:33:58.937 INFO:tasks.rados.rados.0.smithi204.stdout:1992: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-13T11:33:58.962 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1074307 2024-09-13T11:33:59.001 INFO:tasks.rados.rados.0.smithi204.stdout:1997: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-13T11:33:59.007 INFO:tasks.rados.rados.0.smithi204.stdout:1997: writing smithi20438004-14 from 662308 to 1221338 tid 1 2024-09-13T11:33:59.013 INFO:tasks.rados.rados.0.smithi204.stdout:1997: writing smithi20438004-14 from 1632005 to 2235816 tid 2 2024-09-13T11:33:59.020 INFO:tasks.rados.rados.0.smithi204.stdout:1997: writing smithi20438004-14 from 2860044 to 3462308 tid 3 2024-09-13T11:33:59.020 INFO:tasks.rados.rados.0.smithi204.stdout:1988: done (9 left) 2024-09-13T11:33:59.020 INFO:tasks.rados.rados.0.smithi204.stdout:1991: done (8 left) 2024-09-13T11:33:59.020 INFO:tasks.rados.rados.0.smithi204.stdout:1992: done (7 left) 2024-09-13T11:33:59.020 INFO:tasks.rados.rados.0.smithi204.stdout:1993: done (6 left) 2024-09-13T11:33:59.021 INFO:tasks.rados.rados.0.smithi204.stdout:1998: write oid 49 current snap is 0 2024-09-13T11:33:59.021 INFO:tasks.rados.rados.0.smithi204.stdout:1997: finishing write tid 1 to smithi20438004-14 2024-09-13T11:33:59.021 INFO:tasks.rados.rados.0.smithi204.stdout:1998: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-13T11:33:59.027 INFO:tasks.rados.rados.0.smithi204.stdout:1998: writing smithi20438004-49 from 679115 to 1313394 tid 1 2024-09-13T11:33:59.034 INFO:tasks.rados.rados.0.smithi204.stdout:1998: writing smithi20438004-49 from 2090487 to 2554309 tid 2 2024-09-13T11:33:59.037 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:59.035+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:59.039 INFO:tasks.rados.rados.0.smithi204.stdout:1998: writing smithi20438004-49 from 3003820 to 3479115 tid 3 2024-09-13T11:33:59.039 INFO:tasks.rados.rados.0.smithi204.stdout:1999: read oid 28 snap -1 2024-09-13T11:33:59.039 INFO:tasks.rados.rados.0.smithi204.stdout:1999: expect (ObjNum 433 snap 0 seq_num 433) 2024-09-13T11:33:59.039 INFO:tasks.rados.rados.0.smithi204.stdout:2000: read oid 36 snap -1 2024-09-13T11:33:59.040 INFO:tasks.rados.rados.0.smithi204.stdout:2000: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-13T11:33:59.040 INFO:tasks.rados.rados.0.smithi204.stdout:2001: set_chunk oid 7 target oid 2024-09-13T11:33:59.044 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:59.044 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:59.044 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:59.044 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:59.054 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:33:59.137 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:33:59.136+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:33:59.198 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:59.198 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:59.198 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:59.198 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:59.198 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:59.207 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:33:59.238 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:59.238+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:59.312 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:59.312 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.209451682999998, 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016706373, 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.313 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:33:59.314 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.99e-07 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104537+0000", 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034575700000000001 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104552+0000", 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5475999999999999e-05 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.315 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104566+0000", 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3766e-05 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104623+0000", 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7046999999999998e-05 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104650+0000", 2024-09-13T11:33:59.316 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6786000000000001e-05 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117752+0000", 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013102387 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117785+0000", 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2741999999999999e-05 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.317 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.205492356999997, 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025912691000000002, 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:59.318 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105039+0000", 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7100000000000001e-07 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.319 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108050+0000", 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030115910000000001 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108061+0000", 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0512e-05 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108069+0000", 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4479999999999994e-06 2024-09-13T11:33:59.320 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108126+0000", 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7024999999999997e-05 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108165+0000", 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8701000000000001e-05 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.321 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130916+0000", 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022750829 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130951+0000", 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4913999999999998e-05 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:203 3.1 e38/30 3:970ba768:::smithi20438004-4 low_tier:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:59.322 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.202285864000004, 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031827052000000002, 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:33:59.323 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108245+0000", 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1800000000000005e-07 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111202+0000", 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029571440000000001 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.324 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111221+0000", 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8887000000000001e-05 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111248+0000", 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6458e-05 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111707+0000", 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045935799999999999 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.325 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111729+0000", 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140022+0000", 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028292571999999998 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140071+0000", 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9669000000000002e-05 2024-09-13T11:33:59.326 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.199262313999995, 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028930938999999999, 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.327 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111269+0000", 2024-09-13T11:33:59.328 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5700000000000002e-07 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113854+0000", 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025854340000000002 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113866+0000", 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1884e-05 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.329 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113873+0000", 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116175+0000", 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023019 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116218+0000", 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2796000000000002e-05 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.330 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140165+0000", 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023947718999999999 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140199+0000", 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3596000000000002e-05 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:218 3.1 e38/30 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'7, mlcod=38'2)", 2024-09-13T11:33:59.331 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.189399406000007, 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025863384999999999, 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.332 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121132+0000", 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3260000000000001e-06 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124734+0000", 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036019379999999998 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.333 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124748+0000", 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913999999999999e-05 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124760+0000", 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1786e-05 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124849+0000", 2024-09-13T11:33:59.334 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8882999999999996e-05 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124874+0000", 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5466999999999999e-05 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146945+0000", 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022070837999999999 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.335 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146994+0000", 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9233000000000001e-05 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.182466223000006, 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020872166000000001, 2024-09-13T11:33:59.336 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.337 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128066+0000", 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.86e-06 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131456+0000", 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033901360000000002 2024-09-13T11:33:59.338 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131467+0000", 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0806e-05 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131474+0000", 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.339 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131524+0000", 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9262999999999999e-05 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131554+0000", 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0148999999999999e-05 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148883+0000", 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017329445999999998 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.340 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148936+0000", 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2879e-05 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.179028024000004, 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017580887, 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.341 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.342 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131503+0000", 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133995+0000", 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024920630000000001 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134006+0000", 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0696000000000001e-05 2024-09-13T11:33:59.343 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134012+0000", 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136541+0000", 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025286150000000001 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136563+0000", 2024-09-13T11:33:59.344 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2531000000000002e-05 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149031+0000", 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012467951 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149083+0000", 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2040999999999999e-05 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.345 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.152819445999995, 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022274233000000001, 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.346 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157712+0000", 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3689999999999999e-06 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.347 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161974+0000", 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042619160000000001 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161990+0000", 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5506e-05 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161998+0000", 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7520000000000002e-06 2024-09-13T11:33:59.348 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162103+0000", 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104415 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162126+0000", 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3643e-05 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179932+0000", 2024-09-13T11:33:59.349 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017805476000000001 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179985+0000", 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3155999999999997e-05 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.148494213000006, 2024-09-13T11:33:59.350 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018684003000000001, 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.351 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162037+0000", 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.083e-06 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166161+0000", 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041238799999999999 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166179+0000", 2024-09-13T11:33:59.352 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8329999999999999e-05 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166190+0000", 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0822000000000001e-05 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166302+0000", 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011233700000000001 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.353 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166328+0000", 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5185999999999999e-05 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180646+0000", 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014318261000000001 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180720+0000", 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.354 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.144315211000006, 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014657148, 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.355 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1300000000000001e-07 2024-09-13T11:33:59.356 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169754+0000", 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035389779999999999 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169772+0000", 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8020999999999999e-05 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:33:59.357 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0037999999999999e-05 2024-09-13T11:33:59.358 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.358 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169847+0000", 2024-09-13T11:33:59.360 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4695000000000003e-05 2024-09-13T11:33:59.360 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.360 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.360 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.360 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169865+0000", 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7365999999999999e-05 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180834+0000", 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010969016 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180872+0000", 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8621000000000002e-05 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.361 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.132404058000006, 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018123856000000001, 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.362 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178128+0000", 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.363 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181360+0000", 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032324699999999999 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181371+0000", 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0978000000000001e-05 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181378+0000", 2024-09-13T11:33:59.364 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.861e-06 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181431+0000", 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3173000000000001e-05 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181455+0000", 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4236999999999999e-05 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.365 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196213+0000", 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01475807 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196250+0000", 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6636e-05 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:33:59.366 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.129120501000003, 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016397140000000001, 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.367 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181411+0000", 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7399999999999991e-07 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184468+0000", 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030575759999999998 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.368 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184481+0000", 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3072e-05 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184491+0000", 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184550+0000", 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8251999999999997e-05 2024-09-13T11:33:59.369 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184575+0000", 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689000000000001e-05 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197720+0000", 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013144708999999999 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197807+0000", 2024-09-13T11:33:59.370 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6799999999999996e-05 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:281 4.a 4:56ef4fd4:::smithi20438004-32:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.078081788999995, 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018358335999999999, 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.371 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.372 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232449+0000", 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7499999999999998e-07 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232457+0000", 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3610000000000003e-06 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.373 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232470+0000", 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3230999999999999e-05 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232478+0000", 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5989999999999996e-06 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232541+0000", 2024-09-13T11:33:59.374 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3717e-05 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232619+0000", 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248106+0000", 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015486616 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.375 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248991+0000", 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088509199999999998 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249363+0000", 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037176599999999999 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250719+0000", 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001356255 2024-09-13T11:33:59.376 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250765+0000", 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6053000000000001e-05 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250807+0000", 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1675000000000003e-05 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.377 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1696 4.b 4:da92e835:::smithi20438004-50:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.041929973999999, 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016782392, 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1696 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.378 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.379 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268602+0000", 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268608+0000", 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7629999999999997e-06 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268620+0000", 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1950000000000001e-05 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.380 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268632+0000", 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2330999999999999e-05 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269210+0000", 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057761900000000003 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269221+0000", 2024-09-13T11:33:59.381 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.168e-05 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277483+0000", 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0082621180000000006 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277641+0000", 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000157365 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.382 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285383+0000", 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00774195 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1726 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.041861517000001, 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015213417999999999, 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.383 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1726 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:59.384 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.58e-07 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268673+0000", 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7809999999999999e-06 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.385 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268679+0000", 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7729999999999996e-06 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268684+0000", 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269275+0000", 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059124500000000001 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.386 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269281+0000", 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2779999999999999e-06 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281010+0000", 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011728762 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281114+0000", 2024-09-13T11:33:59.387 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103872 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283882+0000", 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027680539999999998 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1727 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:59.388 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 73.041839769999996, 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016698689999999999, 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1727 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:59.389 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268691+0000", 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6100000000000001e-07 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.390 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268694+0000", 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.672e-06 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268699+0000", 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1819999999999997e-06 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268703+0000", 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.391 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269302+0000", 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059838199999999995 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269307+0000", 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3170000000000001e-06 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284218+0000", 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014911308999999999 2024-09-13T11:33:59.392 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284330+0000", 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111715 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285389+0000", 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010592729999999999 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.393 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2018 4.5 4:a9b0c9bd:::smithi20438004-37:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.434872092999996, 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021048396, 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2018 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.394 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875659+0000", 2024-09-13T11:33:59.395 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875661+0000", 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9920000000000002e-06 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875665+0000", 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.396 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875668+0000", 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875714+0000", 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5189999999999999e-05 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875770+0000", 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6824999999999999e-05 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.397 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.896707+0000", 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020936156000000001 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3496 4.18 4:1cf41451:::smithi20438004-7:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 35.556506169999999, 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018591032, 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.398 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3496 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.399 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754026+0000", 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4610000000000001e-06 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754033+0000", 2024-09-13T11:33:59.400 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0369999999999997e-06 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754047+0000", 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4063000000000001e-05 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754056+0000", 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5980000000000003e-06 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.401 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754109+0000", 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2599999999999998e-05 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754165+0000", 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6079999999999998e-05 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.772615+0000", 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018450193 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.402 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4005 4.a 4:56ef4fd4:::smithi20438004-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e81)", 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 24.630857805000002, 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018951355999999999, 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4005 2024-09-13T11:33:59.403 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.404 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679675+0000", 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2079999999999999e-06 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679682+0000", 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3839999999999999e-06 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679701+0000", 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865999999999999e-05 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.405 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679710+0000", 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1980000000000007e-06 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679782+0000", 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2163999999999996e-05 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679869+0000", 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6366000000000006e-05 2024-09-13T11:33:59.406 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.698624+0000", 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018755170000000002 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4055 4.b 4:d738a69d:::smithi20438004-16:head [write 400099~616131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 23.712349334999999, 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01451794, 2024-09-13T11:33:59.407 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4055 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.408 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598185+0000", 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4120000000000003e-06 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598800+0000", 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061453600000000005 2024-09-13T11:33:59.409 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598830+0000", 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0593999999999999e-05 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598847+0000", 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6552e-05 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.410 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598988+0000", 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014121000000000001 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.599001+0000", 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2605e-05 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603641+0000", 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046401250000000002 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.411 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603745+0000", 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010398700000000001 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603921+0000", 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017563 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.607101+0000", 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031806360000000001 2024-09-13T11:33:59.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.611101+0000", 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039995650000000001 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612503+0000", 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014018909999999999 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612653+0000", 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015040499999999999 2024-09-13T11:33:59.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612699+0000", 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5791999999999998e-05 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.414 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:59.416 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:33:59.416 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:33:59.414+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:33:59.517 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:33:59.515+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:33:59.545 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:33:59.570 INFO:tasks.rados.rados.0.smithi204.stdout:2001: set_chunk oid 7 offset: 1331200 length: 1090048 target oid 03a4d2e8be2dfb20f34cd6ff4deb8803599f397d6c740c10c36bbda41701d76b offset: 0 snap -1 2024-09-13T11:33:59.570 INFO:tasks.rados.rados.0.smithi204.stdout:2002: read oid 40 snap -1 2024-09-13T11:33:59.570 INFO:tasks.rados.rados.0.smithi204.stdout:2002: expect deleted 2024-09-13T11:33:59.570 INFO:tasks.rados.rados.0.smithi204.stdout:1997: finishing write tid 2 to smithi20438004-14 2024-09-13T11:33:59.570 INFO:tasks.rados.rados.0.smithi204.stdout:2003: read oid 34 snap -1 2024-09-13T11:33:59.570 INFO:tasks.rados.rados.0.smithi204.stdout:2003: expect (ObjNum 392 snap 0 seq_num 392) 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:2004: read oid 44 snap -1 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:2004: expect deleted 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:2005: read oid 45 snap -1 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:2005: expect (ObjNum 421 snap 0 seq_num 421) 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:2006: delete oid 24 current snap is 0 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: finishing write tid 3 to smithi20438004-14 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: finishing write tid 4 to smithi20438004-14 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: finishing write tid 5 to smithi20438004-14 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: oid 14 updating version 0 to 145 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: oid 14 updating version 145 to 148 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: oid 14 version 148 is already newer than 147 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: oid 14 version 148 is already newer than 146 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 14 v 148 (ObjNum 444 snap 0 seq_num 444) dirty exists 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1997: left oid 14 (ObjNum 444 snap 0 seq_num 444) 2024-09-13T11:33:59.571 INFO:tasks.rados.rados.0.smithi204.stdout:1998: finishing write tid 1 to smithi20438004-49 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: finishing write tid 2 to smithi20438004-49 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: finishing write tid 3 to smithi20438004-49 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: finishing write tid 4 to smithi20438004-49 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: finishing write tid 5 to smithi20438004-49 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: oid 49 updating version 0 to 53 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: oid 49 version 53 is already newer than 51 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: oid 49 updating version 53 to 54 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: oid 49 version 54 is already newer than 52 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 49 v 54 (ObjNum 445 snap 0 seq_num 445) dirty exists 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1998: left oid 49 (ObjNum 445 snap 0 seq_num 445) 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:1999: expect (ObjNum 433 snap 0 seq_num 433) 2024-09-13T11:33:59.572 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:59.573 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:59.574 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:59.574 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:33:59.574 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:59.585 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:33:59.618 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:33:59.616+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:33:59.645 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:33:59.702 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:59.702 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:33:59.703 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:33:59.703 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:33:59.703 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:59.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:33:59.719 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:33:59.718+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:33:59.792 INFO:tasks.rados.rados.0.smithi204.stdout:2000: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-13T11:33:59.817 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:33:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:33:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:33:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:33:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.817 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.723602112999998, 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:59.818 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.819 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.820 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.821 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.822 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.713327546000002, 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.823 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:33:59.824 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:33:59.825 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:33:59.826 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.827 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.700659645000002, 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.829 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.830 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:33:59.831 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.832 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.694049797999995, 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.833 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.834 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.835 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:33:59.836 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.685672768000003, 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:33:59.837 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.839 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:33:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:33:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.840 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:33:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:33:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:33:59.841 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:33:59.842 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.843 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.660541589000005, 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.844 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:33:59.845 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:33:59.846 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.847 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.651640310999994, 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.848 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.849 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.850 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.851 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.852 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.644042588000005, 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.853 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:33:59.854 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.855 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:33:59.856 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.638638658999994, 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:33:59.857 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.858 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.859 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.860 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:33:59.861 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.626014968999996, 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.862 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:33:59.863 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.864 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.865 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:33:59.866 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:59.867 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.611835518999996, 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:59.868 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.869 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:33:59.870 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.871 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.872 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.603617350999997, 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:33:59.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:33:59.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.572793947999997, 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.879 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.881 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:33:59.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:33:59.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:33:59.885 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 92.426079161000004, 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.886 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.890 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 83.720907108999995, 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:33:59.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 82.972954739000002, 2024-09-13T11:33:59.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013151014000000001, 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:33:59.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842045+0000", 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865e-05 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848182+0000", 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061372370000000002 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848205+0000", 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.298e-05 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848218+0000", 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.348e-05 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848300+0000", 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1766000000000003e-05 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848327+0000", 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6446999999999999e-05 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855119+0000", 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067926490000000004 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855177+0000", 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7590000000000003e-05 2024-09-13T11:33:59.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1458 4.e 4:7984cf84:::smithi20438004-46:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 82.787342687000006, 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012793834, 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.900 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1458 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:33:59.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027640+0000", 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.046e-06 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027651+0000", 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.102e-05 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027669+0000", 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027681+0000", 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2177999999999999e-05 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027762+0000", 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0331e-05 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027868+0000", 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106443 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.040432+0000", 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012563632999999999 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1787 4.e 4:7022fc62:::smithi20438004-6:head [read 188928~1265152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 72.541249996000005, 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013752110999999999, 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1787 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.905 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273731+0000", 2024-09-13T11:33:59.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273734+0000", 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3510000000000002e-06 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273741+0000", 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273787+0000", 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6198000000000001e-05 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274432+0000", 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064478300000000001 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274461+0000", 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9051999999999999e-05 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281751+0000", 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007289963 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281984+0000", 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023257099999999999 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.287483+0000", 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054989829999999998 2024-09-13T11:33:59.909 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2762 4.e 4:7984cf84:::smithi20438004-46:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e66)", 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 52.918792578000001, 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022335361000000001, 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.910 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2762 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896189+0000", 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3519999999999999e-06 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896196+0000", 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896210+0000", 2024-09-13T11:33:59.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3439e-05 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896217+0000", 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5889999999999997e-06 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896278+0000", 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1072000000000001e-05 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896338+0000", 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9998999999999999e-05 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.918523+0000", 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022185099999999999 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.914 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3763 4.1 4:8533c921:::smithi20438004-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.681237577000001, 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010966191, 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3763 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133745+0000", 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.75e-06 2024-09-13T11:33:59.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133752+0000", 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133767+0000", 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4577e-05 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133774+0000", 2024-09-13T11:33:59.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133836+0000", 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1787999999999999e-05 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133896+0000", 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9768000000000001e-05 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:33:59.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.144709+0000", 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010813285000000001 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:33:59.919 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:33:59.920 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1861253 2024-09-13T11:33:59.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-13T11:33:59.922 INFO:tasks.ceph.osd.4.smithi038.stderr:2024-09-13T11:33:59.919+0000 7f8f0eb26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33164) UID: 0 2024-09-13T11:33:59.948 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2898844 2024-09-13T11:33:59.948 INFO:tasks.rados.rados.0.smithi204.stdout:1990: got (0) Success 2024-09-13T11:33:59.948 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 118 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-13T11:33:59.948 INFO:tasks.rados.rados.0.smithi204.stdout:1995: expect (ObjNum 408 snap 0 seq_num 408) 2024-09-13T11:34:00.022 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:34:00.021+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:34:00.040 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:34:00.040 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:34:00.040 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:34:00.040 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:34:00.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-13T11:34:00.098 INFO:tasks.rados.rados.0.smithi204.stdout:1994: finishing write tid 1 to smithi20438004-16 2024-09-13T11:34:00.098 INFO:tasks.rados.rados.0.smithi204.stdout:1994: finishing write tid 2 to smithi20438004-16 2024-09-13T11:34:00.098 INFO:tasks.rados.rados.0.smithi204.stdout:1990: done (14 left) 2024-09-13T11:34:00.098 INFO:tasks.rados.rados.0.smithi204.stdout:1995: done (13 left) 2024-09-13T11:34:00.098 INFO:tasks.rados.rados.0.smithi204.stdout:1997: done (12 left) 2024-09-13T11:34:00.098 INFO:tasks.rados.rados.0.smithi204.stdout:1998: done (11 left) 2024-09-13T11:34:00.098 INFO:tasks.rados.rados.0.smithi204.stdout:1999: done (10 left) 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2000: done (9 left) 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2006: done (8 left) 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2007: read oid 3 snap -1 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2007: expect deleted 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2008: tier_promote oid 21 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:1989: got (0) Success 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 23 v 0 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2001: finishing set_chunk to oid 7 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 1331200 length: 1090048 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 0 (ObjNum 435 snap 0 seq_num 435) dirty exists 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:1989: done (9 left) 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2001: done (8 left) 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2009: write oid 17 current snap is 0 2024-09-13T11:34:00.099 INFO:tasks.rados.rados.0.smithi204.stdout:2009: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-13T11:34:00.105 INFO:tasks.rados.rados.0.smithi204.stdout:2009: writing smithi20438004-17 from 712729 to 1497506 tid 1 2024-09-13T11:34:00.111 INFO:tasks.rados.rados.0.smithi204.stdout:2009: writing smithi20438004-17 from 2091098 to 2558589 tid 2 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:2009: writing smithi20438004-17 from 3342313 to 3512729 tid 3 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:2010: tier_promote oid 25 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:1994: finishing write tid 3 to smithi20438004-16 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:1994: finishing write tid 4 to smithi20438004-16 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:1994: finishing write tid 5 to smithi20438004-16 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:1994: oid 16 updating version 0 to 80 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:1994: oid 16 updating version 80 to 81 2024-09-13T11:34:00.117 INFO:tasks.rados.rados.0.smithi204.stdout:1994: oid 16 updating version 81 to 82 2024-09-13T11:34:00.118 INFO:tasks.rados.rados.0.smithi204.stdout:1994: oid 16 updating version 82 to 83 2024-09-13T11:34:00.118 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 16 v 83 (ObjNum 443 snap 0 seq_num 443) dirty exists 2024-09-13T11:34:00.118 INFO:tasks.rados.rados.0.smithi204.stdout:1994: left oid 16 (ObjNum 443 snap 0 seq_num 443) 2024-09-13T11:34:00.118 INFO:tasks.rados.rados.0.smithi204.stdout:1996: got (0) Success 2024-09-13T11:34:00.118 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 94 (ObjNum 366 snap 0 seq_num 366) dirty exists 2024-09-13T11:34:00.118 INFO:tasks.rados.rados.0.smithi204.stdout:2003: expect (ObjNum 392 snap 0 seq_num 392) 2024-09-13T11:34:00.122 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:34:00.121+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:34:00.159 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:34:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:34:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:34:00.159 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:34:00.159 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:34:00.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-13T11:34:00.223 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:34:00.221+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:34:00.277 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:34:00.277 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:34:00.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:34:00.277 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4263.0:138 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'104, mlcod=24'100)", 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 114.537481012, 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067196920000000002, 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:34:00.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737180+0000", 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.68e-07 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737343+0000", 2024-09-13T11:34:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016218800000000001 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737351+0000", 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0749999999999998e-06 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737355+0000", 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7949999999999998e-06 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737398+0000", 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.244e-05 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.737420+0000", 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.215e-05 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743871+0000", 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064505400000000003 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:05.743899+0000", 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8836e-05 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:9 4.9 4:94e25a0d:::smithi20438004-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.797555384999995, 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012557795, 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.282 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 9 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477105+0000", 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477107+0000", 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6729999999999999e-06 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477756+0000", 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064862600000000002 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477769+0000", 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3224000000000001e-05 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477776+0000", 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6990000000000003e-06 2024-09-13T11:34:00.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477886+0000", 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010976699999999999 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.477891+0000", 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9579999999999998e-06 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480278+0000", 2024-09-13T11:34:00.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023866410000000001 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480377+0000", 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9469000000000006e-05 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.480485+0000", 2024-09-13T11:34:00.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107755 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488517+0000", 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080317389999999995 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489071+0000", 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055442999999999996 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489540+0000", 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046886600000000002 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489603+0000", 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2514999999999994e-05 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489663+0000", 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0433000000000002e-05 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:10 4.9 4:94e25a0d:::smithi20438004-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.792496826999994, 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078173280000000001, 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.290 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 10 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482164+0000", 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482165+0000", 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.114e-06 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482554+0000", 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000389157 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482566+0000", 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2332e-05 2024-09-13T11:34:00.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482573+0000", 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3659999999999997e-06 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486865+0000", 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042924149999999999 2024-09-13T11:34:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.486958+0000", 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3257999999999994e-05 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.487116+0000", 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015787 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488525+0000", 2024-09-13T11:34:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409106 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489326+0000", 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080097199999999995 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489853+0000", 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052625600000000001 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489929+0000", 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5918000000000001e-05 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.489981+0000", 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2564e-05 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:11 4.9 4:94e25a0d:::smithi20438004-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.792073005000006, 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090593110000000004, 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 11 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482588+0000", 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1500000000000005e-07 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482591+0000", 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.202e-06 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482597+0000", 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1719999999999998e-06 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482600+0000", 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5559999999999999e-06 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488053+0000", 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054530799999999999 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488146+0000", 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3071000000000004e-05 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488240+0000", 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4071000000000001e-05 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488886+0000", 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00064589699999999999 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.490035+0000", 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011487660000000001 2024-09-13T11:34:00.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491561+0000", 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015258820000000001 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491599+0000", 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8631000000000003e-05 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491647+0000", 2024-09-13T11:34:00.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7468e-05 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:12 4.9 4:94e25a0d:::smithi20438004-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.792054961000005, 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093243159999999992, 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.303 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 12 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482606+0000", 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1399999999999998e-07 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482608+0000", 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4899999999999999e-06 2024-09-13T11:34:00.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482611+0000", 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6210000000000001e-06 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.482615+0000", 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.749e-06 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488448+0000", 2024-09-13T11:34:00.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005833057 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.488497+0000", 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8918999999999997e-05 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491707+0000", 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00320996 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491796+0000", 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9463999999999996e-05 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.491930+0000", 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013374200000000001 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:252 4.9 4:94e25a0d:::smithi20438004-3:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.308 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.075814188999999, 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020934432999999999, 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 252 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198846+0000", 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198848+0000", 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0030000000000001e-06 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198856+0000", 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3080000000000002e-06 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198873+0000", 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6937000000000001e-05 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198881+0000", 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7889999999999995e-06 2024-09-13T11:34:00.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198938+0000", 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7630000000000002e-05 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.198994+0000", 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5622999999999999e-05 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:34:00.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.217970+0000", 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976208000000001 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218550+0000", 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058039799999999998 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.218994+0000", 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044406699999999998 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219678+0000", 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068345400000000001 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219735+0000", 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6743000000000002e-05 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219781+0000", 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6273000000000002e-05 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:257 4.9 4:9441befd:::smithi20438004-8:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.061973399999999, 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017912915000000001, 2024-09-13T11:34:00.315 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 257 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212687+0000", 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212689+0000", 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7260000000000001e-06 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212702+0000", 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2539e-05 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212718+0000", 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6399e-05 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212729+0000", 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0711e-05 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212793+0000", 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000006e-05 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.212867+0000", 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3409000000000001e-05 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228564+0000", 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015697283999999999 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229414+0000", 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085038699999999997 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229572+0000", 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015754399999999999 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230530+0000", 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095766600000000003 2024-09-13T11:34:00.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230567+0000", 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7270000000000001e-05 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.230600+0000", 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3290000000000001e-05 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3068 3.1 e72/51 3:ba0ff2fc:::35d746116e015b742d2355e1d921a21f28a06eaf655b8cfcf0782553e2cf9a5a:head v 72'267, mlcod=70'265)", 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 44.658051141000001, 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073641619999999996, 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.616610+0000", 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620678+0000", 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040678880000000004 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620694+0000", 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.611e-05 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620705+0000", 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1124999999999999e-05 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620768+0000", 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2626000000000007e-05 2024-09-13T11:34:00.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.620794+0000", 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6446000000000001e-05 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623920+0000", 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003126298 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:15.623974+0000", 2024-09-13T11:34:00.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3375000000000003e-05 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3600 4.9 4:94e25a0d:::smithi20438004-3:head [write 527819~624406,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.123551673000001, 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076023949999999996, 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.326 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3600 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151109+0000", 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151112+0000", 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7070000000000001e-06 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151819+0000", 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00070706700000000003 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151834+0000", 2024-09-13T11:34:00.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4944e-05 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151841+0000", 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1760000000000004e-06 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151897+0000", 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5948e-05 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.151966+0000", 2024-09-13T11:34:00.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9283999999999996e-05 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7,14", 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.152076+0000", 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109633 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154405+0000", 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023291950000000001 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.154749+0000", 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034383100000000001 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158638+0000", 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038894250000000002 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158677+0000", 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9167e-05 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.158711+0000", 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4017999999999999e-05 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:3987 4.b e81/52 4:da92e835:::smithi20438004-50:head v 81'69, mlcod=76'67)", 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.065196419999999, 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068559900000000002, 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.332 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.209464+0000", 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7100000000000001e-07 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210320+0000", 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085537199999999997 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210332+0000", 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.224e-05 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210343+0000", 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0549e-05 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210424+0000", 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1209000000000001e-05 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.210450+0000", 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6673e-05 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216271+0000", 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058204060000000002 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.216320+0000", 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9370000000000003e-05 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4013 4.9 4:9441befd:::smithi20438004-8:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e81)", 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.580613885999998, 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017394468999999999, 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4013 2024-09-13T11:34:00.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694047+0000", 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694048+0000", 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.026e-06 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694055+0000", 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9249999999999998e-06 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694065+0000", 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9049999999999992e-06 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694070+0000", 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4419999999999997e-06 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694114+0000", 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4341000000000001e-05 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.694170+0000", 2024-09-13T11:34:00.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5618999999999997e-05 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:34.711441+0000", 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017271211000000002 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4060 4.15 4:a9b0c9bd:::smithi20438004-37:head [write 1366083~511456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.659875016000001, 2024-09-13T11:34:00.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073320010000000003, 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4060 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614786+0000", 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614787+0000", 2024-09-13T11:34:00.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9500000000000001e-07 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615260+0000", 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047298899999999999 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615270+0000", 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0597999999999999e-05 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615276+0000", 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.524e-06 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615325+0000", 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9530000000000002e-05 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615404+0000", 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9208000000000005e-05 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615492+0000", 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7163999999999994e-05 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.617037+0000", 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001545374 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.619988+0000", 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029509470000000002 2024-09-13T11:34:00.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622041+0000", 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002052824 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622091+0000", 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9932000000000002e-05 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622118+0000", 2024-09-13T11:34:00.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7016e-05 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4061 4.15 4:a9b0c9bd:::smithi20438004-37:head [setxattr _header (56) in=7b,truncate 2016906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.659372054999999, 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069574540000000001, 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.348 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4061 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:34:00.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615289+0000", 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0200000000000003e-07 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615291+0000", 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2299999999999998e-06 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615296+0000", 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9790000000000001e-06 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615299+0000", 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2040000000000002e-06 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616228+0000", 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000928213 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616282+0000", 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4147999999999997e-05 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,10", 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616369+0000", 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7595000000000004e-05 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.617083+0000", 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071378200000000004 2024-09-13T11:34:00.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622163+0000", 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050794780000000001 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622189+0000", 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6539000000000001e-05 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622216+0000", 2024-09-13T11:34:00.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6443000000000001e-05 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622246+0000", 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0440999999999999e-05 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4062 4.15 4:a9b0c9bd:::smithi20438004-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.659338301999998, 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070487859999999996, 2024-09-13T11:34:00.354 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4062 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615322+0000", 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615323+0000", 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615326+0000", 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:34:00.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615334+0000", 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9300000000000003e-06 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.615342+0000", 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4610000000000002e-06 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616576+0000", 2024-09-13T11:34:00.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001234787 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.616622+0000", 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549e-05 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622271+0000", 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056493400000000001 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622313+0000", 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1616999999999998e-05 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.622371+0000", 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8252999999999999e-05 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4680 4.1b 4:de617177:::smithi20438004-44:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:54.139096+0000", 2024-09-13T11:34:00.359 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1355647949999996, 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073709869999999999, 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4680 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.139096+0000", 2024-09-13T11:34:00.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.139096+0000", 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.139098+0000", 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2929999999999999e-06 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.139905+0000", 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080733900000000002 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.139918+0000", 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2143999999999999e-05 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.139923+0000", 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4149999999999998e-06 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.139992+0000", 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8693000000000001e-05 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.140053+0000", 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1272000000000006e-05 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,6,10,11", 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.140162+0000", 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108872 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.141665+0000", 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001503306 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.141709+0000", 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4223e-05 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.144044+0000", 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023348689999999998 2024-09-13T11:34:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.144234+0000", 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018932199999999999 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.146351+0000", 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021174969999999999 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.146390+0000", 2024-09-13T11:34:00.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8470000000000003e-05 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:54.146467+0000", 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7272000000000001e-05 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4926 4.b 4:d738a69d:::smithi20438004-16:head [write 645501~483781,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e99)", 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:59.552300+0000", 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.72236047699999995, 2024-09-13T11:34:00.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014650847, 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4926 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.552300+0000", 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.552300+0000", 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.552301+0000", 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.189e-06 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.553448+0000", 2024-09-13T11:34:00.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001146705 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.553462+0000", 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4348000000000001e-05 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.553483+0000", 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0358999999999999e-05 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554893+0000", 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001409914 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554907+0000", 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3983e-05 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.560496+0000", 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005589147 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.560612+0000", 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011639200000000001 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,11,13", 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.560732+0000", 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119758 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566404+0000", 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056724619999999996 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566420+0000", 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5693999999999999e-05 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566719+0000", 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029928600000000001 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566822+0000", 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103007 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566875+0000", 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2942000000000002e-05 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566951+0000", 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5661000000000006e-05 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4927 4.b 4:d738a69d:::smithi20438004-16:head [write 1573523~460153] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-13T11:34:00.373 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:59.553509+0000", 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.72115194400000004, 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014194910999999999, 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4927 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.553509+0000", 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.553509+0000", 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.553509+0000", 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.44e-07 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554076+0000", 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056660200000000003 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554088+0000", 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2367e-05 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554094+0000", 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0039999999999996e-06 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554943+0000", 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00084839500000000005 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554950+0000", 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6559999999999992e-06 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.563044+0000", 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080934600000000002 2024-09-13T11:34:00.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.563144+0000", 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100373 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,11,13", 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.563311+0000", 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000166995 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566659+0000", 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033474580000000002 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566678+0000", 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9279e-05 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567554+0000", 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087586900000000004 2024-09-13T11:34:00.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567600+0000", 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5901e-05 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567636+0000", 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6477000000000003e-05 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567704+0000", 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7430999999999999e-05 2024-09-13T11:34:00.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4928 4.b 4:d738a69d:::smithi20438004-16:head [write 2548974~763384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:59.554106+0000", 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.72055463099999995, 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016679913000000001, 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4928 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554106+0000", 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554106+0000", 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554107+0000", 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4499999999999997e-07 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554807+0000", 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00070087600000000002 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554824+0000", 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6152999999999999e-05 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554833+0000", 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2480000000000002e-06 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554984+0000", 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000151495 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554993+0000", 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4169999999999992e-06 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.564405+0000", 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094125960000000005 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.564527+0000", 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121204 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,11,13", 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.564658+0000", 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000131846 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.566769+0000", 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002110949 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.568450+0000", 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016805920000000001 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.569190+0000", 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073995199999999999 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.570646+0000", 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001456578 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.570706+0000", 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9970999999999999e-05 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.570786+0000", 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9491000000000002e-05 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4929 4.b 4:d738a69d:::smithi20438004-16:head [setxattr _header (56) in=7b,truncate 3445501] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-13T11:34:00.390 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:59.554878+0000", 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.71978261499999996, 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016303515000000001, 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4929 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554878+0000", 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554878+0000", 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554879+0000", 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.001e-06 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554888+0000", 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3899999999999999e-06 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554901+0000", 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2354000000000001e-05 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554910+0000", 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5570000000000001e-06 2024-09-13T11:34:00.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.555022+0000", 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111197 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.555028+0000", 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8109999999999997e-06 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567010+0000", 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011982059999999999 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567130+0000", 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119857 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,11,13", 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567259+0000", 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012858800000000001 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.568144+0000", 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00088563400000000005 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.569250+0000", 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001105331 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.570468+0000", 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001217692 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.571073+0000", 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060579099999999995 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.571112+0000", 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8435e-05 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.571182+0000", 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9816999999999999e-05 2024-09-13T11:34:00.397 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4930 4.b 4:d738a69d:::smithi20438004-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:59.554939+0000", 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.71972117400000002, 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016432763999999999, 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.398 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4930 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554939+0000", 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554939+0000", 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554940+0000", 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5799999999999998e-07 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554944+0000", 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.766e-06 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554952+0000", 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3480000000000004e-06 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.554962+0000", 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2520000000000005e-06 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.555072+0000", 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109978 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.555083+0000", 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1705999999999999e-05 2024-09-13T11:34:00.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567735+0000", 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012652022000000001 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.567791+0000", 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5399999999999998e-05 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.571215+0000", 2024-09-13T11:34:00.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034238179999999999 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.571262+0000", 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6995e-05 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:59.571372+0000", 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110621 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.403 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.404 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:34:00.405 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1953385 2024-09-13T11:34:00.405 INFO:tasks.rados.rados.0.smithi204.stdout:2005: expect (ObjNum 421 snap 0 seq_num 421) 2024-09-13T11:34:00.405 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3009803 2024-09-13T11:34:00.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-13T11:34:00.406 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:34:00.404+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:34:00.506 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:34:00.504+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:34:00.553 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:34:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:34:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:34:00.553 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:34:00.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-13T11:34:00.607 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:34:00.606+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:34:00.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-13T11:34:00.613 INFO:tasks.rados.rados.0.smithi204.stdout:2008: got (0) Success 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 21 v 50 (ObjNum 430 snap 0 seq_num 430) dirty exists 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:1994: done (9 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:1996: done (8 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2002: done (7 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2003: done (6 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2004: done (5 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2005: done (4 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2007: done (3 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2008: done (2 left) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2011: read oid 49 snap -1 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2011: expect (ObjNum 445 snap 0 seq_num 445) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2012: read oid 7 snap -1 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2012: expect (ObjNum 435 snap 0 seq_num 435) 2024-09-13T11:34:00.614 INFO:tasks.rados.rados.0.smithi204.stdout:2013: tier_flush oid 48 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2013: tier_flush oid 48 snap -1 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2009: finishing write tid 1 to smithi20438004-17 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2009: finishing write tid 2 to smithi20438004-17 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2014: tier_flush oid 40 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2014: tier_flush oid 40 snap -1 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2010: got (0) Success 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 148 (ObjNum 439 snap 0 seq_num 439) dirty exists 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2009: finishing write tid 3 to smithi20438004-17 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2009: finishing write tid 4 to smithi20438004-17 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2009: finishing write tid 5 to smithi20438004-17 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2009: oid 17 updating version 0 to 88 2024-09-13T11:34:00.615 INFO:tasks.rados.rados.0.smithi204.stdout:2009: oid 17 version 88 is already newer than 87 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:2009: oid 17 updating version 88 to 90 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:2009: oid 17 version 90 is already newer than 89 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 17 v 90 (ObjNum 447 snap 0 seq_num 447) dirty exists 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:2009: left oid 17 (ObjNum 447 snap 0 seq_num 447) 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:2009: done (5 left) 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:2010: done (4 left) 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:2015: write oid 45 current snap is 0 2024-09-13T11:34:00.616 INFO:tasks.rados.rados.0.smithi204.stdout:2015: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-13T11:34:00.620 INFO:tasks.rados.rados.0.smithi204.stdout:2015: writing smithi20438004-45 from 729536 to 1189562 tid 1 2024-09-13T11:34:00.626 INFO:tasks.rados.rados.0.smithi204.stdout:2015: writing smithi20438004-45 from 1749580 to 2360729 tid 2 2024-09-13T11:34:00.632 INFO:tasks.rados.rados.0.smithi204.stdout:2015: writing smithi20438004-45 from 2853383 to 3305833 tid 3 2024-09-13T11:34:00.632 INFO:tasks.rados.rados.0.smithi204.stdout:2016: write oid 18 current snap is 0 2024-09-13T11:34:00.632 INFO:tasks.rados.rados.0.smithi204.stdout:2016: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-09-13T11:34:00.640 INFO:tasks.rados.rados.0.smithi204.stdout:2016: writing smithi20438004-18 from 746343 to 1397971 tid 1 2024-09-13T11:34:00.646 INFO:tasks.rados.rados.0.smithi204.stdout:2016: writing smithi20438004-18 from 1924415 to 2679222 tid 2 2024-09-13T11:34:00.651 INFO:tasks.rados.rados.0.smithi204.stdout:2016: writing smithi20438004-18 from 3397159 to 3546343 tid 3 2024-09-13T11:34:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:2017: read oid 4 snap -1 2024-09-13T11:34:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:2017: expect deleted 2024-09-13T11:34:00.652 INFO:tasks.rados.rados.0.smithi204.stdout:2012: expect (ObjNum 435 snap 0 seq_num 435) 2024-09-13T11:34:00.662 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 918763 2024-09-13T11:34:00.673 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:34:00.673 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:34:00.673 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-09-13T11:34:00.673 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-09-13T11:34:00.673 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:34:00.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-13T11:34:00.728 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:34:00.726+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:34:00.757 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:34:00.758 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:34:00.780 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:34:00.780 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:34:00.780 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:34:00.799 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:34:00.799 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:34:00.806 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1026366 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:25 4.8 4:1cf41451:::smithi20438004-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:34:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 94.283886832999997, 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013758497, 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:34:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522663+0000", 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.522665+0000", 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523390+0000", 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072434500000000005 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523403+0000", 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2948e-05 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523411+0000", 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6109999999999994e-06 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523488+0000", 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6403999999999996e-05 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.523496+0000", 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1159999999999999e-06 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525476+0000", 2024-09-13T11:34:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019807470000000001 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525660+0000", 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018323600000000001 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.525852+0000", 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019269299999999999 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533793+0000", 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079404720000000005 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.533850+0000", 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7569999999999999e-05 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536313+0000", 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024627210000000002 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536367+0000", 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3748000000000001e-05 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:26.536422+0000", 2024-09-13T11:34:00.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5124999999999998e-05 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:256 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.594948692000003, 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015539183, 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.818 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 256 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211602+0000", 2024-09-13T11:34:00.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211603+0000", 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211611+0000", 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7400000000000004e-06 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211629+0000", 2024-09-13T11:34:00.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7589e-05 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211636+0000", 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5580000000000003e-06 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211712+0000", 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5921999999999996e-05 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.211822+0000", 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110134 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220265+0000", 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084424819999999994 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220863+0000", 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059827999999999997 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.221258+0000", 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000394738 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227082+0000", 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058237269999999999 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227114+0000", 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2588e-05 2024-09-13T11:34:00.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.227141+0000", 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6667999999999999e-05 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.826 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:262 4.8 4:1251e6cc:::smithi20438004-13:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.586606493999994, 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011146559, 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 262 2024-09-13T11:34:00.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219944+0000", 2024-09-13T11:34:00.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219946+0000", 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.804e-06 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219952+0000", 2024-09-13T11:34:00.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8900000000000004e-06 2024-09-13T11:34:00.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219974+0000", 2024-09-13T11:34:00.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2065999999999998e-05 2024-09-13T11:34:00.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.219983+0000", 2024-09-13T11:34:00.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6709999999999994e-06 2024-09-13T11:34:00.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220596+0000", 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061274300000000001 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.220682+0000", 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6154000000000002e-05 2024-09-13T11:34:00.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228213+0000", 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075309169999999998 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.228907+0000", 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069417999999999997 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.229020+0000", 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113047 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231019+0000", 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019986240000000001 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231066+0000", 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7132000000000002e-05 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.231090+0000", 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:294 4.8 4:1d367c64:::smithi20438004-45:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.566248490999996, 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012993378, 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 294 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240302+0000", 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240303+0000", 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.223e-06 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240309+0000", 2024-09-13T11:34:00.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.976e-06 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240324+0000", 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5135e-05 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240334+0000", 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0308e-05 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240391+0000", 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6379999999999999e-05 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.240442+0000", 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0730999999999999e-05 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248573+0000", 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081319019999999999 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249195+0000", 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062111299999999996 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.249314+0000", 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119288 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253193+0000", 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038794839999999999 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253250+0000", 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6369000000000002e-05 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.253295+0000", 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5469000000000001e-05 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:306 4.8 4:1cf41451:::smithi20438004-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.558447732000005, 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010265013999999999, 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 306 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248103+0000", 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248104+0000", 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9e-06 2024-09-13T11:34:00.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248116+0000", 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1724e-05 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248142+0000", 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6101999999999998e-05 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248157+0000", 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4547e-05 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248297+0000", 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000140638 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.248399+0000", 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101523 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251251+0000", 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002851887 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.251867+0000", 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061572199999999997 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.252191+0000", 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032463400000000002 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258306+0000", 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061150010000000001 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258335+0000", 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8608000000000001e-05 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.258368+0000", 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2728000000000002e-05 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:795 4.8 4:1cf41451:::smithi20438004-7:head [write 473964~611066] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 93.008940924000001, 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015457046, 2024-09-13T11:34:00.848 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 795 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797609+0000", 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.797612+0000", 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807707+0000", 2024-09-13T11:34:00.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010094537000000001 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807724+0000", 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6991e-05 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807732+0000", 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807805+0000", 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2324999999999996e-05 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.807893+0000", 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8373000000000002e-05 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.808016+0000", 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012304899999999999 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.810782+0000", 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002765656 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812805+0000", 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002023062 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812856+0000", 2024-09-13T11:34:00.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1434000000000003e-05 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.812999+0000", 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014307999999999999 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:27.813066+0000", 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7214999999999999e-05 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1054 4.8 4:1d367c64:::smithi20438004-45:head [write 578174~533948,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 90.046752389000005, 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010776299, 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.855 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759798+0000", 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.759801+0000", 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5499999999999999e-06 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761478+0000", 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016765860000000001 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761497+0000", 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9160999999999998e-05 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761510+0000", 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2874000000000001e-05 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761568+0000", 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8254999999999997e-05 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.761575+0000", 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2580000000000001e-06 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763172+0000", 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015973179999999999 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763294+0000", 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012246499999999999 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.763486+0000", 2024-09-13T11:34:00.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019193500000000001 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770155+0000", 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669108 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770199+0000", 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3686000000000002e-05 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770453+0000", 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025428400000000002 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770545+0000", 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1679000000000001e-05 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:30.770574+0000", 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9139999999999999e-05 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1485 4.8 4:1251e6cc:::smithi20438004-13:head [read 0~3966419,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 83.696285293000003, 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01527358, 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1485 2024-09-13T11:34:00.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110265+0000", 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2300000000000002e-07 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110269+0000", 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3400000000000002e-06 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110274+0000", 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2349999999999999e-06 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.110278+0000", 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6100000000000002e-06 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117718+0000", 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074403120000000001 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.117821+0000", 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103501 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:37.125539+0000", 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077171590000000003 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1636 4.8 4:1d367c64:::smithi20438004-45:head [read 0~3378174,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 79.806118126000001, 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091050230000000003, 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1636 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000432+0000", 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000433+0000", 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1200000000000003e-07 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000436+0000", 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0860000000000002e-06 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000443+0000", 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8619999999999997e-06 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000447+0000", 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8800000000000001e-06 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.001976+0000", 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001529774 2024-09-13T11:34:00.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.002101+0000", 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124824 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009537+0000", 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074360850000000003 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1637 4.8 4:1d367c64:::smithi20438004-45:head [sparse-read 0~2368645] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 79.806095596999995, 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010163363, 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1637 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.871 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:34:00.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000455+0000", 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9400000000000001e-07 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000457+0000", 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7570000000000001e-06 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000460+0000", 2024-09-13T11:34:00.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3720000000000001e-06 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.000463+0000", 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.225e-06 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009568+0000", 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091047630000000001 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.009626+0000", 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8142e-05 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:41.010618+0000", 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099181000000000009 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:1749 4.8 4:1d367c64:::smithi20438004-45:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:00.875 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.018849431000007, 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49694346099999998, 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1749 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787701+0000", 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787703+0000", 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6840000000000001e-06 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787708+0000", 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.609e-06 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787721+0000", 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2405000000000001e-05 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787728+0000", 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.391e-06 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787780+0000", 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1897000000000001e-05 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:45.787845+0000", 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5184000000000005e-05 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:32:46.284644+0000", 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.49679929099999998 2024-09-13T11:34:00.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3085 4.d 4:b15c1843:::smithi20438004-33:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+full_force+supports_pool_eio e72)", 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 44.531974011000003, 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012088290999999999, 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.880 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3085 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274576+0000", 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274578+0000", 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0609999999999998e-06 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274588+0000", 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9119999999999993e-06 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274602+0000", 2024-09-13T11:34:00.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3655000000000001e-05 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.274613+0000", 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1199000000000001e-05 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.275921+0000", 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013075000000000001 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:34:00.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.275933+0000", 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1905999999999999e-05 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280509+0000", 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045760560000000002 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280683+0000", 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017461199999999999 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.280858+0000", 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017475999999999999 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.284147+0000", 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032893900000000001 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.284277+0000", 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012992 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286340+0000", 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020622779999999999 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286614+0000", 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274486 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:16.286665+0000", 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0556000000000002e-05 2024-09-13T11:34:00.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3500 4.d 4:bca378c8:::smithi20438004-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.047613587000001, 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026796718000000001, 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.887 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3500 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758937+0000", 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758939+0000", 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6929999999999999e-06 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758952+0000", 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2337000000000001e-05 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.758985+0000", 2024-09-13T11:34:00.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3099000000000003e-05 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759005+0000", 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0197e-05 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759114+0000", 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010864799999999999 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.759208+0000", 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4728999999999994e-05 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785733+0000", 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026525014999999999 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3505 4.14 4:2c957d44:::smithi20438004-22:head [write 1940712~656106] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:34:00.891 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.031218793000001, 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010057587999999999, 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3505 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775332+0000", 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.775333+0000", 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.488e-06 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776349+0000", 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010159909999999999 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776360+0000", 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1481000000000001e-05 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776369+0000", 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5429999999999994e-06 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776417+0000", 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8182e-05 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776474+0000", 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6669000000000002e-05 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.776581+0000", 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010673199999999999 2024-09-13T11:34:00.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.778281+0000", 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001700811 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785203+0000", 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069212960000000004 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785249+0000", 2024-09-13T11:34:00.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6448e-05 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785290+0000", 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1340000000000001e-05 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:23.785389+0000", 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8607e-05 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3568 4.d 4:b15c1843:::smithi20438004-33:head [write 477398~798659,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 35.521408307999998, 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010217238999999999, 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.898 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3568 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285142+0000", 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.285148+0000", 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8449999999999999e-06 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286131+0000", 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00098294599999999995 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286146+0000", 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4712000000000001e-05 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286153+0000", 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6399999999999997e-06 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286206+0000", 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3284e-05 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286271+0000", 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4921999999999999e-05 2024-09-13T11:34:00.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.286382+0000", 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011042799999999999 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.288353+0000", 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019708669999999998 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.288371+0000", 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8538000000000001e-05 2024-09-13T11:34:00.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295281+0000", 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069098720000000001 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295321+0000", 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0145000000000003e-05 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:25.295359+0000", 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8040000000000002e-05 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3632 4.8 4:1251e6cc:::smithi20438004-13:head [read 618102~2928175,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.575574193999998, 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010153435000000001, 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3632 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230976+0000", 2024-09-13T11:34:00.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230977+0000", 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9200000000000001e-07 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230984+0000", 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8990000000000004e-06 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230991+0000", 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5089999999999996e-06 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.230998+0000", 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3999999999999997e-06 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.235260+0000", 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042621330000000004 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.235319+0000", 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9549999999999997e-05 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:26.241130+0000", 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058103520000000004 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:3839 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:34:00.908 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.233932706000001, 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023869665000000002, 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3839 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572618+0000", 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572620+0000", 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0339999999999999e-06 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572630+0000", 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0056e-05 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572658+0000", 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8756e-05 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572673+0000", 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572789+0000", 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011540900000000001 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.572850+0000", 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1397999999999997e-05 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:30.596487+0000", 2024-09-13T11:34:00.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023637162 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4631.0:4056 4.b e82/52 4:d738a69d:::smithi20438004-16:head v 82'74, mlcod=81'71)", 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:35.605551+0000", 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.200999096, 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090727229999999996, 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605551+0000", 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605551+0000", 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.605552+0000", 2024-09-13T11:34:00.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3099999999999997e-07 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606348+0000", 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079557499999999997 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606356+0000", 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8659999999999995e-06 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606369+0000", 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2302e-05 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606446+0000", 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7441999999999995e-05 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.606468+0000", 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1440000000000001e-05 2024-09-13T11:34:00.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614588+0000", 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081204710000000006 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:35.614624+0000", 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5896000000000003e-05 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4084 4.d 4:bca378c8:::smithi20438004-20:head [write 433713~482982] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:36.199349+0000", 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.60720152, 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097880520000000002, 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4084 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.199349+0000", 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.199349+0000", 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.199351+0000", 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9379999999999999e-06 2024-09-13T11:34:00.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.201928+0000", 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025777249999999999 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.201951+0000", 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2248000000000001e-05 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.201963+0000", 2024-09-13T11:34:00.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2092e-05 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.202041+0000", 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7782999999999997e-05 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.202127+0000", 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6254999999999993e-05 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-09-13T11:34:00.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.202243+0000", 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116477 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.204044+0000", 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018004749999999999 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.204444+0000", 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040071200000000001 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.208902+0000", 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044577870000000004 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.209103+0000", 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000200865 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:36.209137+0000", 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3695000000000002e-05 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.923 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4631.0:4497 4.1c 4.de2cd65c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-09-13T11:33:48.158495+0000", 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.648055701000001, 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.32777995100000001, 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "continuous": false, 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4631", 2024-09-13T11:34:00.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 4497 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158495+0000", 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158495+0000", 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158496+0000", 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4900000000000004e-07 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158512+0000", 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6209e-05 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158526+0000", 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3798000000000001e-05 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158533+0000", 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4229999999999999e-06 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158597+0000", 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4256000000000003e-05 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.158605+0000", 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5839999999999997e-06 2024-09-13T11:34:00.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.486261+0000", 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.32765663 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-09-13T11:33:48.486275+0000", 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3101999999999999e-05 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-09-13T11:34:00.928 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-09-13T11:34:00.929 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:34:00.930 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:34:00.808+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:34:00.930 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:34:00.930 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:34:00.930 INFO:tasks.rados.rados.0.smithi204.stdout:2015: finishing write tid 1 to smithi20438004-45 2024-09-13T11:34:00.931 INFO:tasks.rados.rados.0.smithi204.stdout:2014: got (2) No such file or directory 2024-09-13T11:34:00.931 INFO:tasks.rados.rados.0.smithi204.stdout:2014: got expected ENOENT (src dne) 2024-09-13T11:34:00.931 INFO:tasks.rados.rados.0.smithi204.stdout:2011: expect (ObjNum 445 snap 0 seq_num 445) 2024-09-13T11:34:00.931 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:00.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-13T11:34:00.932 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:34:00.932 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:34:00.933 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:34:00.933 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:34:00.933 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:34:00.934 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:34:00.934 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:34:00.934 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:34:00.934 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:34:00.935 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:34:00.935 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:34:00.935 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:34:00.935 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:34:00.928+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:34:00.936 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:34:00.936 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:34:00.937 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:34:00.954 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:34:00.954 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:34:00.956 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2460184 2024-09-13T11:34:00.973 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:34:00.973 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:34:00.991 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:34:00.991 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:34:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:34:00.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:34:01.031 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:34:01.029+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:34:01.037 INFO:tasks.rados.rados.0.smithi204.stdout:2015: finishing write tid 2 to smithi20438004-45 2024-09-13T11:34:01.037 INFO:tasks.rados.rados.0.smithi204.stdout:2015: finishing write tid 3 to smithi20438004-45 2024-09-13T11:34:01.037 INFO:tasks.rados.rados.0.smithi204.stdout:2015: finishing write tid 4 to smithi20438004-45 2024-09-13T11:34:01.037 INFO:tasks.rados.rados.0.smithi204.stdout:2015: finishing write tid 5 to smithi20438004-45 2024-09-13T11:34:01.037 INFO:tasks.rados.rados.0.smithi204.stdout:2015: oid 45 updating version 0 to 101 2024-09-13T11:34:01.037 INFO:tasks.rados.rados.0.smithi204.stdout:2015: oid 45 version 101 is already newer than 100 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2015: oid 45 version 101 is already newer than 99 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2015: oid 45 updating version 101 to 102 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 102 (ObjNum 448 snap 0 seq_num 448) dirty exists 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2015: left oid 45 (ObjNum 448 snap 0 seq_num 448) 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2016: finishing write tid 1 to smithi20438004-18 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2018: read oid 43 snap -1 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2018: expect (ObjNum 432 snap 0 seq_num 432) 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2011: done (7 left) 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2012: done (6 left) 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2014: done (5 left) 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2015: done (4 left) 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2019: write (excl) oid 2 current snap is 0 2024-09-13T11:34:01.038 INFO:tasks.rados.rados.0.smithi204.stdout:2019: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-13T11:34:01.044 INFO:tasks.rados.rados.0.smithi204.stdout:2019: writing smithi20438004-2 from 763150 to 1490027 tid 1 2024-09-13T11:34:01.045 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.045 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.050 INFO:tasks.rados.rados.0.smithi204.stdout:2019: writing smithi20438004-2 from 1982897 to 2597715 tid 2 2024-09-13T11:34:01.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-13T11:34:01.056 INFO:tasks.rados.rados.0.smithi204.stdout:2019: writing smithi20438004-2 from 3024582 to 3499576 tid 3 2024-09-13T11:34:01.056 INFO:tasks.rados.rados.0.smithi204.stdout:2020: read oid 21 snap -1 2024-09-13T11:34:01.056 INFO:tasks.rados.rados.0.smithi204.stdout:2020: expect (ObjNum 430 snap 0 seq_num 430) 2024-09-13T11:34:01.056 INFO:tasks.rados.rados.0.smithi204.stdout:2021: read oid 17 snap -1 2024-09-13T11:34:01.056 INFO:tasks.rados.rados.0.smithi204.stdout:2021: expect (ObjNum 447 snap 0 seq_num 447) 2024-09-13T11:34:01.056 INFO:tasks.rados.rados.0.smithi204.stdout:2022: set_chunk oid 7 target oid 2024-09-13T11:34:01.132 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:34:01.131+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:34:01.160 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.161 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-13T11:34:01.232 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:34:01.231+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:34:01.276 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.276 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-13T11:34:01.293 INFO:tasks.rados.rados.0.smithi204.stdout:2022: set_chunk oid 7 offset: 2342400 length: 51200 target oid 1c274e7ae7d6636a17cd41f101f4f10df50612df4589ecd282a509d78f872da9 offset: 0 snap -1 2024-09-13T11:34:01.293 INFO:tasks.rados.rados.0.smithi204.stdout:2023: read oid 38 snap -1 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2023: expect (ObjNum 427 snap 0 seq_num 427) 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2024: write oid 34 current snap is 0 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: finishing write tid 2 to smithi20438004-18 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: finishing write tid 3 to smithi20438004-18 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: finishing write tid 4 to smithi20438004-18 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: finishing write tid 5 to smithi20438004-18 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: oid 18 updating version 0 to 164 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: oid 18 updating version 164 to 165 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: oid 18 version 165 is already newer than 163 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:2016: oid 18 version 165 is already newer than 162 2024-09-13T11:34:01.294 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 18 v 165 (ObjNum 449 snap 0 seq_num 449) dirty exists 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2016: left oid 18 (ObjNum 449 snap 0 seq_num 449) 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: finishing write tid 1 to smithi20438004-2 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: finishing write tid 2 to smithi20438004-2 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: finishing write tid 3 to smithi20438004-2 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: finishing write tid 4 to smithi20438004-2 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: finishing write tid 5 to smithi20438004-2 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: oid 2 updating version 0 to 101 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: oid 2 updating version 101 to 102 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: oid 2 updating version 102 to 103 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: oid 2 updating version 103 to 104 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 2 v 104 (ObjNum 450 snap 0 seq_num 450) dirty exists 2024-09-13T11:34:01.295 INFO:tasks.rados.rados.0.smithi204.stdout:2019: left oid 2 (ObjNum 450 snap 0 seq_num 450) 2024-09-13T11:34:01.296 INFO:tasks.rados.rados.0.smithi204.stdout:2021: expect (ObjNum 447 snap 0 seq_num 447) 2024-09-13T11:34:01.296 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:01.333 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:34:01.332+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:34:01.399 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2838345 2024-09-13T11:34:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-13T11:34:01.417 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:01.454 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:34:01.453+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:34:01.482 INFO:tasks.rados.rados.0.smithi204.stdout:2020: expect (ObjNum 430 snap 0 seq_num 430) 2024-09-13T11:34:01.510 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 3068056 2024-09-13T11:34:01.511 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:01.514 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.514 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-13T11:34:01.534 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-13T11:34:01.533+0000 7fc30aa60640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32883) UID: 0 2024-09-13T11:34:01.568 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:34:01.589 INFO:tasks.rados.rados.0.smithi204.stdout:2023: expect (ObjNum 427 snap 0 seq_num 427) 2024-09-13T11:34:01.621 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2529334 2024-09-13T11:34:01.635 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:34:01.633+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:34:01.664 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.664 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.673 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-13T11:34:01.736 INFO:tasks.ceph.osd.3.smithi204.stderr:2024-09-13T11:34:01.735+0000 7fd8926e9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33336) UID: 0 2024-09-13T11:34:01.776 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.776 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.785 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-13T11:34:01.827 INFO:tasks.rados.rados.0.smithi204.stdout:2022: finishing set_chunk to oid 7 2024-09-13T11:34:01.827 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 7 v 98 (ObjNum 435 snap 0 seq_num 435) dirty exists 2024-09-13T11:34:01.827 INFO:tasks.rados.rados.0.smithi204.stdout:2018: expect (ObjNum 432 snap 0 seq_num 432) 2024-09-13T11:34:01.827 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:01.836 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:34:01.835+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:34:01.888 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:01.888 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:01.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-13T11:34:01.937 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:34:01.935+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:34:02.000 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.000 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-13T11:34:02.009 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 658074 2024-09-13T11:34:02.039 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:34:02.037+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:34:02.090 INFO:tasks.rados.rados.0.smithi204.stdout:2013: got (0) Success 2024-09-13T11:34:02.090 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-13T11:34:02.090 INFO:tasks.rados.rados.0.smithi204.stdout:2024: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-13T11:34:02.096 INFO:tasks.rados.rados.0.smithi204.stdout:2024: writing smithi20438004-34 from 779957 to 1182083 tid 1 2024-09-13T11:34:02.104 INFO:tasks.rados.rados.0.smithi204.stdout:2024: writing smithi20438004-34 from 1925026 to 2683502 tid 2 2024-09-13T11:34:02.109 INFO:tasks.rados.rados.0.smithi204.stdout:2024: writing smithi20438004-34 from 3335652 to 3579957 tid 3 2024-09-13T11:34:02.109 INFO:tasks.rados.rados.0.smithi204.stdout:2013: done (9 left) 2024-09-13T11:34:02.109 INFO:tasks.rados.rados.0.smithi204.stdout:2016: done (8 left) 2024-09-13T11:34:02.109 INFO:tasks.rados.rados.0.smithi204.stdout:2017: done (7 left) 2024-09-13T11:34:02.109 INFO:tasks.rados.rados.0.smithi204.stdout:2018: done (6 left) 2024-09-13T11:34:02.109 INFO:tasks.rados.rados.0.smithi204.stdout:2019: done (5 left) 2024-09-13T11:34:02.110 INFO:tasks.rados.rados.0.smithi204.stdout:2020: done (4 left) 2024-09-13T11:34:02.110 INFO:tasks.rados.rados.0.smithi204.stdout:2021: done (3 left) 2024-09-13T11:34:02.110 INFO:tasks.rados.rados.0.smithi204.stdout:2022: done (2 left) 2024-09-13T11:34:02.110 INFO:tasks.rados.rados.0.smithi204.stdout:2023: done (1 left) 2024-09-13T11:34:02.110 INFO:tasks.rados.rados.0.smithi204.stdout:2025: write (excl) oid 22 current snap is 0 2024-09-13T11:34:02.110 INFO:tasks.rados.rados.0.smithi204.stdout:2025: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-13T11:34:02.111 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.111 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.116 INFO:tasks.rados.rados.0.smithi204.stdout:2025: writing smithi20438004-22 from 796764 to 1274139 tid 1 2024-09-13T11:34:02.120 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-13T11:34:02.123 INFO:tasks.rados.rados.0.smithi204.stdout:2025: writing smithi20438004-22 from 1983508 to 2601995 tid 2 2024-09-13T11:34:02.129 INFO:tasks.rados.rados.0.smithi204.stdout:2025: writing smithi20438004-22 from 3363075 to 3596764 tid 3 2024-09-13T11:34:02.129 INFO:tasks.rados.rados.0.smithi204.stdout:2026: write (excl) oid 11 current snap is 0 2024-09-13T11:34:02.129 INFO:tasks.rados.rados.0.smithi204.stdout:2024: finishing write tid 1 to smithi20438004-34 2024-09-13T11:34:02.129 INFO:tasks.rados.rados.0.smithi204.stdout:2026: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-13T11:34:02.134 INFO:tasks.rados.rados.0.smithi204.stdout:2026: writing smithi20438004-11 from 413571 to 966195 tid 1 2024-09-13T11:34:02.139 INFO:tasks.ceph.osd.10.smithi154.stderr:2024-09-13T11:34:02.138+0000 7fbc001cc640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32886) UID: 0 2024-09-13T11:34:02.142 INFO:tasks.rados.rados.0.smithi204.stdout:2026: writing smithi20438004-11 from 1641990 to 2404135 tid 2 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2026: writing smithi20438004-11 from 2990498 to 3415661 tid 3 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2027: tier_flush oid 33 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2027: tier_flush oid 33 snap -1 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: finishing write tid 2 to smithi20438004-34 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: finishing write tid 3 to smithi20438004-34 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: finishing write tid 4 to smithi20438004-34 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: finishing write tid 5 to smithi20438004-34 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: oid 34 updating version 0 to 166 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: oid 34 updating version 166 to 169 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: oid 34 version 169 is already newer than 168 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:2024: oid 34 version 169 is already newer than 167 2024-09-13T11:34:02.149 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 34 v 169 (ObjNum 451 snap 0 seq_num 451) dirty exists 2024-09-13T11:34:02.150 INFO:tasks.rados.rados.0.smithi204.stdout:2024: left oid 34 (ObjNum 451 snap 0 seq_num 451) 2024-09-13T11:34:02.150 INFO:tasks.rados.rados.0.smithi204.stdout:2025: finishing write tid 1 to smithi20438004-22 2024-09-13T11:34:02.150 INFO:tasks.rados.rados.0.smithi204.stdout:2025: finishing write tid 2 to smithi20438004-22 2024-09-13T11:34:02.150 INFO:tasks.rados.rados.0.smithi204.stdout:2024: done (3 left) 2024-09-13T11:34:02.150 INFO:tasks.rados.rados.0.smithi204.stdout:2028: set_chunk oid 48 target oid 2024-09-13T11:34:02.152 INFO:tasks.rados.rados.0.smithi204.stdout:2028: set_chunk oid 48 offset: 977920 length: 131072 target oid 6ec34eb0d37a8c0ccc91ae346cce97bbfa179c0c235961e9c020ac811f232936 offset: 0 snap -1 2024-09-13T11:34:02.152 INFO:tasks.rados.rados.0.smithi204.stdout:2029: write oid 19 current snap is 0 2024-09-13T11:34:02.153 INFO:tasks.rados.rados.0.smithi204.stdout:2025: finishing write tid 3 to smithi20438004-22 2024-09-13T11:34:02.153 INFO:tasks.rados.rados.0.smithi204.stdout:2029: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-13T11:34:02.158 INFO:tasks.rados.rados.0.smithi204.stdout:2029: writing smithi20438004-19 from 430378 to 1058251 tid 1 2024-09-13T11:34:02.165 INFO:tasks.rados.rados.0.smithi204.stdout:2029: writing smithi20438004-19 from 1700472 to 2322628 tid 2 2024-09-13T11:34:02.173 INFO:tasks.rados.rados.0.smithi204.stdout:2029: writing smithi20438004-19 from 3017921 to 3630378 tid 3 2024-09-13T11:34:02.173 INFO:tasks.rados.rados.0.smithi204.stdout:2030: write (excl) oid 3 current snap is 0 2024-09-13T11:34:02.173 INFO:tasks.rados.rados.0.smithi204.stdout:2025: finishing write tid 4 to smithi20438004-22 2024-09-13T11:34:02.173 INFO:tasks.rados.rados.0.smithi204.stdout:2025: finishing write tid 5 to smithi20438004-22 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:2025: oid 22 updating version 0 to 125 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:2025: oid 22 updating version 125 to 128 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:2025: oid 22 version 128 is already newer than 127 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:2025: oid 22 version 128 is already newer than 126 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 128 (ObjNum 452 snap 0 seq_num 452) dirty exists 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:2025: left oid 22 (ObjNum 452 snap 0 seq_num 452) 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:2026: finishing write tid 1 to smithi20438004-11 2024-09-13T11:34:02.174 INFO:tasks.rados.rados.0.smithi204.stdout:2030: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-13T11:34:02.179 INFO:tasks.rados.rados.0.smithi204.stdout:2030: writing smithi20438004-3 from 447185 to 1150307 tid 1 2024-09-13T11:34:02.186 INFO:tasks.rados.rados.0.smithi204.stdout:2030: writing smithi20438004-3 from 1642601 to 2408415 tid 2 2024-09-13T11:34:02.191 INFO:tasks.rados.rados.0.smithi204.stdout:2030: writing smithi20438004-3 from 2928991 to 3376698 tid 3 2024-09-13T11:34:02.192 INFO:tasks.rados.rados.0.smithi204.stdout:2025: done (5 left) 2024-09-13T11:34:02.192 INFO:tasks.rados.rados.0.smithi204.stdout:2031: set_chunk oid 8 target oid 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2031: set_chunk oid 8 offset: 1708544 length: 467456 target oid 3b4ed34ebbc0135d730341f622ef86354afb5eb947afa7e1b332ff36f9d156aa offset: 0 snap -1 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2032: read oid 6 snap -1 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2032: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2033: read oid 14 snap -1 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2033: expect (ObjNum 444 snap 0 seq_num 444) 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2034: read oid 23 snap -1 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2034: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2035: write oid 31 current snap is 0 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2026: finishing write tid 2 to smithi20438004-11 2024-09-13T11:34:02.200 INFO:tasks.rados.rados.0.smithi204.stdout:2035: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-09-13T11:34:02.206 INFO:tasks.rados.rados.0.smithi204.stdout:2035: writing smithi20438004-31 from 463992 to 1242363 tid 1 2024-09-13T11:34:02.212 INFO:tasks.rados.rados.0.smithi204.stdout:2035: writing smithi20438004-31 from 1701083 to 2326908 tid 2 2024-09-13T11:34:02.220 INFO:tasks.rados.rados.0.smithi204.stdout:2035: writing smithi20438004-31 from 2956414 to 3615393 tid 3 2024-09-13T11:34:02.220 INFO:tasks.rados.rados.0.smithi204.stdout:2036: write (excl) oid 45 current snap is 0 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: finishing write tid 3 to smithi20438004-11 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: finishing write tid 4 to smithi20438004-11 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: finishing write tid 5 to smithi20438004-11 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: oid 11 updating version 0 to 150 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: oid 11 updating version 150 to 153 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: oid 11 version 153 is already newer than 152 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: oid 11 version 153 is already newer than 151 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 11 v 153 (ObjNum 453 snap 0 seq_num 453) dirty exists 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2026: left oid 11 (ObjNum 453 snap 0 seq_num 453) 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2028: finishing set_chunk to oid 48 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:Range is overlapped: oid 48 set_chunk returned error code -95 offset: 977920 length: 131072 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 48 v 0 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-13T11:34:02.221 INFO:tasks.rados.rados.0.smithi204.stdout:2029: finishing write tid 1 to smithi20438004-19 2024-09-13T11:34:02.222 INFO:tasks.rados.rados.0.smithi204.stdout:2029: finishing write tid 2 to smithi20438004-19 2024-09-13T11:34:02.222 INFO:tasks.rados.rados.0.smithi204.stdout:2027: got (0) Success 2024-09-13T11:34:02.222 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 33 v 0 (ObjNum 338 snap 0 seq_num 338) dirty exists 2024-09-13T11:34:02.222 INFO:tasks.rados.rados.0.smithi204.stdout:2029: finishing write tid 3 to smithi20438004-19 2024-09-13T11:34:02.222 INFO:tasks.rados.rados.0.smithi204.stdout:2036: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-13T11:34:02.225 INFO:tasks.rados.rados.0.smithi204.stdout:2036: writing smithi20438004-45 from 480799 to 1050772 tid 1 2024-09-13T11:34:02.226 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.227 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.232 INFO:tasks.rados.rados.0.smithi204.stdout:2036: writing smithi20438004-45 from 1475918 to 2245401 tid 2 2024-09-13T11:34:02.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2036: writing smithi20438004-45 from 2700190 to 3286794 tid 3 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2026: done (10 left) 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2027: done (9 left) 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2028: done (8 left) 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2037: read oid 9 snap -1 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2037: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2038: read oid 26 snap -1 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2038: expect (ObjNum 438 snap 0 seq_num 438) 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2039: write oid 36 current snap is 0 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2029: finishing write tid 4 to smithi20438004-19 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2029: finishing write tid 5 to smithi20438004-19 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2029: oid 19 updating version 0 to 153 2024-09-13T11:34:02.239 INFO:tasks.rados.rados.0.smithi204.stdout:2029: oid 19 version 153 is already newer than 151 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2029: oid 19 version 153 is already newer than 152 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2029: oid 19 version 153 is already newer than 150 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 19 v 153 (ObjNum 454 snap 0 seq_num 454) dirty exists 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2029: left oid 19 (ObjNum 454 snap 0 seq_num 454) 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: finishing write tid 1 to smithi20438004-3 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: finishing write tid 2 to smithi20438004-3 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: finishing write tid 3 to smithi20438004-3 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: finishing write tid 4 to smithi20438004-3 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: finishing write tid 5 to smithi20438004-3 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: oid 3 updating version 0 to 68 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: oid 3 updating version 68 to 70 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: oid 3 version 70 is already newer than 69 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:2030: oid 3 version 70 is already newer than 67 2024-09-13T11:34:02.240 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 3 v 70 (ObjNum 455 snap 0 seq_num 455) dirty exists 2024-09-13T11:34:02.241 INFO:tasks.rados.rados.0.smithi204.stdout:2030: left oid 3 (ObjNum 455 snap 0 seq_num 455) 2024-09-13T11:34:02.241 INFO:tasks.rados.rados.0.smithi204.stdout:2039: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-13T11:34:02.242 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:34:02.241+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:34:02.244 INFO:tasks.rados.rados.0.smithi204.stdout:2039: writing smithi20438004-36 from 497606 to 1142828 tid 1 2024-09-13T11:34:02.252 INFO:tasks.rados.rados.0.smithi204.stdout:2039: writing smithi20438004-36 from 1934400 to 2563894 tid 2 2024-09-13T11:34:02.259 INFO:tasks.rados.rados.0.smithi204.stdout:2039: writing smithi20438004-36 from 3243966 to 3697606 tid 3 2024-09-13T11:34:02.259 INFO:tasks.rados.rados.0.smithi204.stdout:2029: done (10 left) 2024-09-13T11:34:02.259 INFO:tasks.rados.rados.0.smithi204.stdout:2030: done (9 left) 2024-09-13T11:34:02.259 INFO:tasks.rados.rados.0.smithi204.stdout:2040: read oid 24 snap -1 2024-09-13T11:34:02.259 INFO:tasks.rados.rados.0.smithi204.stdout:2040: expect deleted 2024-09-13T11:34:02.260 INFO:tasks.rados.rados.0.smithi204.stdout:2041: read oid 19 snap -1 2024-09-13T11:34:02.260 INFO:tasks.rados.rados.0.smithi204.stdout:2041: expect (ObjNum 454 snap 0 seq_num 454) 2024-09-13T11:34:02.260 INFO:tasks.rados.rados.0.smithi204.stdout:2042: tier_promote oid 20 2024-09-13T11:34:02.260 INFO:tasks.rados.rados.0.smithi204.stdout:2031: finishing set_chunk to oid 8 2024-09-13T11:34:02.260 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 8 v 66 (ObjNum 441 snap 0 seq_num 441) dirty exists 2024-09-13T11:34:02.260 INFO:tasks.rados.rados.0.smithi204.stdout:2031: done (11 left) 2024-09-13T11:34:02.260 INFO:tasks.rados.rados.0.smithi204.stdout:2043: set_chunk oid 25 target oid 2024-09-13T11:34:02.278 INFO:tasks.rados.rados.0.smithi204.stdout:2043: set_chunk oid 25 offset: 1426944 length: 730112 target oid 7d6691982e890b79f323d91b922c95a74b6fafce80f3d3662261e4c76ce080c7 offset: 0 snap -1 2024-09-13T11:34:02.278 INFO:tasks.rados.rados.0.smithi204.stdout:2044: read oid 30 snap -1 2024-09-13T11:34:02.279 INFO:tasks.rados.rados.0.smithi204.stdout:2044: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-13T11:34:02.279 INFO:tasks.rados.rados.0.smithi204.stdout:2045: write oid 4 current snap is 0 2024-09-13T11:34:02.279 INFO:tasks.rados.rados.0.smithi204.stdout:2045: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-13T11:34:02.285 INFO:tasks.rados.rados.0.smithi204.stdout:2045: writing smithi20438004-4 from 514413 to 1234884 tid 1 2024-09-13T11:34:02.292 INFO:tasks.rados.rados.0.smithi204.stdout:2045: writing smithi20438004-4 from 1876529 to 2649681 tid 2 2024-09-13T11:34:02.298 INFO:tasks.rados.rados.0.smithi204.stdout:2045: writing smithi20438004-4 from 3438683 to 3714413 tid 3 2024-09-13T11:34:02.298 INFO:tasks.rados.rados.0.smithi204.stdout:2046: read oid 11 snap -1 2024-09-13T11:34:02.298 INFO:tasks.rados.rados.0.smithi204.stdout:2046: expect (ObjNum 453 snap 0 seq_num 453) 2024-09-13T11:34:02.298 INFO:tasks.rados.rados.0.smithi204.stdout:2047: write (excl) oid 1 current snap is 0 2024-09-13T11:34:02.298 INFO:tasks.rados.rados.0.smithi204.stdout:2047: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-13T11:34:02.305 INFO:tasks.rados.rados.0.smithi204.stdout:2047: writing smithi20438004-1 from 531220 to 1326940 tid 1 2024-09-13T11:34:02.312 INFO:tasks.rados.rados.0.smithi204.stdout:2047: writing smithi20438004-1 from 1935011 to 2451821 tid 2 2024-09-13T11:34:02.317 INFO:tasks.rados.rados.0.smithi204.stdout:2047: writing smithi20438004-1 from 3066106 to 3486526 tid 3 2024-09-13T11:34:02.318 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 16 2024-09-13T11:34:02.318 INFO:tasks.rados.rados.0.smithi204.stdout:2032: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-13T11:34:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.339 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.343 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:34:02.341+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:34:02.349 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-13T11:34:02.394 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1876759 2024-09-13T11:34:02.425 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 652578 2024-09-13T11:34:02.444 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:34:02.443+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:34:02.451 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.451 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-13T11:34:02.468 INFO:tasks.rados.rados.0.smithi204.stdout:2034: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-13T11:34:02.474 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1213124 2024-09-13T11:34:02.518 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:02.545 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:34:02.543+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:34:02.547 INFO:tasks.rados.rados.0.smithi204.stdout:2033: expect (ObjNum 444 snap 0 seq_num 444) 2024-09-13T11:34:02.547 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:02.562 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.562 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.563 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 13] out_osds: [1, 5, 9] dead_osds: [2, 12] live_osds: [0, 4, 8, 1, 5, 9, 13, 6, 10, 14, 3, 7, 11, 15] 2024-09-13T11:34:02.563 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-13T11:34:02.563 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-13T11:34:02.563 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-13T11:34:02.563 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-13T11:34:02.563 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-13T11:34:02.563 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-09-13T11:34:02.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-13T11:34:02.574 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:34:02.646 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:34:02.644+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:34:02.667 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:02.667 INFO:teuthology.orchestra.run.smithi204.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-13T11:34:02.667 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:02.673 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.673 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-13T11:34:02.738 INFO:tasks.rados.rados.0.smithi204.stdout:2035: finishing write tid 1 to smithi20438004-31 2024-09-13T11:34:02.738 INFO:tasks.rados.rados.0.smithi204.stdout:2035: finishing write tid 2 to smithi20438004-31 2024-09-13T11:34:02.738 INFO:tasks.rados.rados.0.smithi204.stdout:2032: done (15 left) 2024-09-13T11:34:02.738 INFO:tasks.rados.rados.0.smithi204.stdout:2033: done (14 left) 2024-09-13T11:34:02.738 INFO:tasks.rados.rados.0.smithi204.stdout:2034: done (13 left) 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2048: write oid 22 current snap is 0 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: finishing write tid 3 to smithi20438004-31 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: finishing write tid 4 to smithi20438004-31 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: finishing write tid 5 to smithi20438004-31 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: oid 31 updating version 0 to 75 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: oid 31 version 75 is already newer than 74 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: oid 31 version 75 is already newer than 73 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: oid 31 updating version 75 to 76 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 31 v 76 (ObjNum 456 snap 0 seq_num 456) dirty exists 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2035: left oid 31 (ObjNum 456 snap 0 seq_num 456) 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2036: finishing write tid 1 to smithi20438004-45 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2036: finishing write tid 2 to smithi20438004-45 2024-09-13T11:34:02.739 INFO:tasks.rados.rados.0.smithi204.stdout:2048: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-13T11:34:02.743 INFO:tasks.rados.rados.0.smithi204.stdout:2048: writing smithi20438004-22 from 548027 to 1018996 tid 1 2024-09-13T11:34:02.746 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:34:02.745+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:34:02.751 INFO:tasks.rados.rados.0.smithi204.stdout:2048: writing smithi20438004-22 from 1593493 to 2370314 tid 2 2024-09-13T11:34:02.759 INFO:tasks.rados.rados.0.smithi204.stdout:2048: writing smithi20438004-22 from 3093529 to 3725221 tid 3 2024-09-13T11:34:02.759 INFO:tasks.rados.rados.0.smithi204.stdout:2035: done (13 left) 2024-09-13T11:34:02.759 INFO:tasks.rados.rados.0.smithi204.stdout:2049: delete oid 44 current snap is 0 2024-09-13T11:34:02.759 INFO:tasks.rados.rados.0.smithi204.stdout:2036: finishing write tid 3 to smithi20438004-45 2024-09-13T11:34:02.759 INFO:tasks.rados.rados.0.smithi204.stdout:2036: finishing write tid 4 to smithi20438004-45 2024-09-13T11:34:02.759 INFO:tasks.rados.rados.0.smithi204.stdout:2036: finishing write tid 5 to smithi20438004-45 2024-09-13T11:34:02.759 INFO:tasks.rados.rados.0.smithi204.stdout:2036: oid 45 updating version 0 to 104 2024-09-13T11:34:02.760 INFO:tasks.rados.rados.0.smithi204.stdout:2036: oid 45 updating version 104 to 105 2024-09-13T11:34:02.760 INFO:tasks.rados.rados.0.smithi204.stdout:2036: oid 45 updating version 105 to 106 2024-09-13T11:34:02.760 INFO:tasks.rados.rados.0.smithi204.stdout:2036: oid 45 updating version 106 to 107 2024-09-13T11:34:02.760 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 45 v 107 (ObjNum 457 snap 0 seq_num 457) dirty exists 2024-09-13T11:34:02.760 INFO:tasks.rados.rados.0.smithi204.stdout:2036: left oid 45 (ObjNum 457 snap 0 seq_num 457) 2024-09-13T11:34:02.760 INFO:tasks.rados.rados.0.smithi204.stdout:2038: expect (ObjNum 438 snap 0 seq_num 438) 2024-09-13T11:34:02.790 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.790 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-13T11:34:02.847 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:34:02.845+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:34:02.902 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:02.902 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:02.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-13T11:34:02.923 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 965153 2024-09-13T11:34:02.953 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:34:02.951+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:34:02.987 INFO:tasks.rados.rados.0.smithi204.stdout:2037: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-13T11:34:02.987 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:02.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-13T11:34:03.048 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:34:03.046+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:34:03.054 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:03.054 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:03.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-13T11:34:03.130 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-09-13T11:34:03.130 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-13T11:34:03.148 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-09-13T11:34:03.148 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-13T11:34:03.148 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T11:34:03.149 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:34:03.147+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:34:03.152 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 1575476 2024-09-13T11:34:03.165 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-09-13T11:34:03.165 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-13T11:34:03.168 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-13T11:34:03.168 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-13T11:34:03.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-13T11:34:03.181 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-09-13T11:34:03.181 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-13T11:34:03.196 INFO:tasks.rados.rados.0.smithi204.stdout:2039: finishing write tid 1 to smithi20438004-36 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2039: finishing write tid 2 to smithi20438004-36 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2036: done (13 left) 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2037: done (12 left) 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2038: done (11 left) 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2049: done (10 left) 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 10 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2042: got (2) No such file or directory 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2042: got expected ENOENT (src dne) 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 20 v 111 (ObjNum 129 snap 0 seq_num 3593823008) clean dne 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2039: finishing write tid 3 to smithi20438004-36 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2040: done (9 left) 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout:2042: done (8 left) 2024-09-13T11:34:03.197 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 8 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2039: finishing write tid 4 to smithi20438004-36 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2039: finishing write tid 5 to smithi20438004-36 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2039: oid 36 updating version 0 to 75 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2039: oid 36 updating version 75 to 76 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2039: oid 36 version 76 is already newer than 73 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2039: oid 36 version 76 is already newer than 74 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 36 v 76 (ObjNum 458 snap 0 seq_num 458) dirty exists 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2039: left oid 36 (ObjNum 458 snap 0 seq_num 458) 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:2041: expect (ObjNum 454 snap 0 seq_num 454) 2024-09-13T11:34:03.198 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:03.199 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-09-13T11:34:03.199 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-13T11:34:03.215 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-09-13T11:34:03.215 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-13T11:34:03.232 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-09-13T11:34:03.232 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-13T11:34:03.248 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8: {} 2024-09-13T11:34:03.248 INFO:teuthology.orchestra.run.smithi038.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-13T11:34:03.249 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:34:03.247+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:34:03.264 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 0 2024-09-13T11:34:03.265 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9: {} 2024-09-13T11:34:03.265 INFO:teuthology.orchestra.run.smithi038.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-13T11:34:03.281 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:34:03.282 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:34:03.282 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:34:03.282 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:34:03.282 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:34:03.282 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10: {} 2024-09-13T11:34:03.283 INFO:teuthology.orchestra.run.smithi038.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-13T11:34:03.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-13T11:34:03.298 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11: {} 2024-09-13T11:34:03.298 INFO:teuthology.orchestra.run.smithi038.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-13T11:34:03.299 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.12 2024-09-13T11:34:03.315 INFO:teuthology.orchestra.run.smithi038.stdout:osd.13: {} 2024-09-13T11:34:03.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-13T11:34:03.332 INFO:teuthology.orchestra.run.smithi038.stdout:osd.14: {} 2024-09-13T11:34:03.332 INFO:teuthology.orchestra.run.smithi038.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-13T11:34:03.349 INFO:teuthology.orchestra.run.smithi038.stdout:osd.15: {} 2024-09-13T11:34:03.349 INFO:teuthology.orchestra.run.smithi038.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-13T11:34:03.349 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:34:03.347+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:34:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-13T11:34:03.352 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-13T11:34:03.395 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:34:03.395 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:34:03.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:34:03.395 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:34:03.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.395 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:254 4.0 4:04aeeab9:::smithi20438004-5:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 96.191086356, 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022108323999999999, 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 254 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.396 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201593+0000", 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201594+0000", 2024-09-13T11:34:03.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5990000000000001e-06 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201603+0000", 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.568e-06 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201619+0000", 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6540000000000001e-05 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201629+0000", 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3200000000000006e-06 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201685+0000", 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6078000000000001e-05 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.201747+0000", 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2242999999999993e-05 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220103+0000", 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01835548 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.220937+0000", 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083429299999999995 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.221938+0000", 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010010710000000001 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223567+0000", 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001628846 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223638+0000", 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1165000000000001e-05 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.223701+0000", 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3120999999999994e-05 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:285 4.0 4:0c9026a3:::smithi20438004-36:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 96.158101896999995, 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012191677, 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.402 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 285 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234577+0000", 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234578+0000", 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7400000000000002e-07 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234583+0000", 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9110000000000001e-06 2024-09-13T11:34:03.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234593+0000", 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234598+0000", 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9509999999999997e-06 2024-09-13T11:34:03.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234639+0000", 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1826999999999998e-05 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.234688+0000", 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9039000000000003e-05 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 7,9", 2024-09-13T11:34:03.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239203+0000", 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045143739999999998 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.239979+0000", 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077664100000000001 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.242634+0000", 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026544899999999998 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246683+0000", 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040491499999999996 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246723+0000", 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0429000000000002e-05 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.246769+0000", 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5430999999999999e-05 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.408 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1715 4.f 4:fea8c5b2:::smithi20438004-39:head [write 769787~704463,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.124292956999994, 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020210576000000001, 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.409 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1715 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268386+0000", 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.268388+0000", 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4750000000000001e-06 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269172+0000", 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078478 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269184+0000", 2024-09-13T11:34:03.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.182e-05 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269194+0000", 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8570000000000007e-06 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269269+0000", 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4795000000000005e-05 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269283+0000", 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4005000000000001e-05 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275426+0000", 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061428050000000003 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275540+0000", 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011432199999999999 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.275658+0000", 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118071 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278252+0000", 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594224 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279142+0000", 2024-09-13T11:34:03.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088964600000000001 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288321+0000", 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091785600000000005 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288363+0000", 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2957000000000002e-05 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288492+0000", 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012896899999999999 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288597+0000", 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010429000000000001 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.416 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1716 4.f 4:fea8c5b2:::smithi20438004-39:head [write 2267961~101826] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.123466100000002, 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019610079999999998, 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1716 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269213+0000", 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269214+0000", 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9699999999999995e-07 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269305+0000", 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1330999999999997e-05 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269314+0000", 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9209999999999998e-06 2024-09-13T11:34:03.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269321+0000", 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1330000000000001e-06 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269363+0000", 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2388999999999997e-05 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269372+0000", 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6070000000000008e-06 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277175+0000", 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078032580000000004 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277247+0000", 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1773999999999994e-05 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.277354+0000", 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010704000000000001 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288308+0000", 2024-09-13T11:34:03.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010953353000000001 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288328+0000", 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0353e-05 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288640+0000", 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031178999999999999 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288706+0000", 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6749999999999996e-05 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288765+0000", 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8876999999999999e-05 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288823+0000", 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7806999999999997e-05 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.424 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1717 4.f 4:fea8c5b2:::smithi20438004-39:head [setxattr _header (56) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.123341838000002, 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019744788999999999, 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.425 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1717 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269337+0000", 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269338+0000", 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5199999999999996e-07 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269342+0000", 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.924e-06 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269349+0000", 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9619999999999996e-06 2024-09-13T11:34:03.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269355+0000", 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1210000000000003e-06 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269399+0000", 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4360000000000002e-05 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5570000000000001e-06 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278293+0000", 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088867549999999997 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278372+0000", 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9561999999999995e-05 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 12,13,14", 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.278492+0000", 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119654 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288335+0000", 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098426840000000008 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288672+0000", 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033785899999999998 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288863+0000", 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019105599999999999 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288904+0000", 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0738000000000002e-05 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288949+0000", 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4443000000000001e-05 2024-09-13T11:34:03.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289082+0000", 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013346199999999999 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1718 4.f 4:fea8c5b2:::smithi20438004-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.123312364, 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019946973, 2024-09-13T11:34:03.432 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1718 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269367+0000", 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8999999999999996e-07 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269373+0000", 2024-09-13T11:34:03.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8629999999999996e-06 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269379+0000", 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4979999999999997e-06 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269383+0000", 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7260000000000002e-06 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269443+0000", 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9616e-05 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269451+0000", 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7600000000000002e-06 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279055+0000", 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009604586 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279104+0000", 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8131999999999999e-05 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289119+0000", 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010015616 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289177+0000", 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8060999999999999e-05 2024-09-13T11:34:03.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289314+0000", 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000136525 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1732 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.123287868999995, 2024-09-13T11:34:03.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017069699000000001, 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1732 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269391+0000", 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269392+0000", 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269395+0000", 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-06 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269400+0000", 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.198e-06 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269406+0000", 2024-09-13T11:34:03.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9410000000000004e-06 2024-09-13T11:34:03.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269514+0000", 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108124 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269524+0000", 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7589999999999992e-06 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279170+0000", 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096457209999999995 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.279248+0000", 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8147999999999998e-05 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286461+0000", 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072129789999999996 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1733 4.f 4:f653a832:::smithi20438004-19:head [read 0~3613472,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:03.446 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.123244893000006, 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0187283, 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1733 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.447 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:34:03.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269434+0000", 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269435+0000", 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7800000000000001e-07 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269438+0000", 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.535e-06 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269447+0000", 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6559999999999998e-06 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269452+0000", 2024-09-13T11:34:03.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.972e-06 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269552+0000", 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9820000000000003e-05 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269556+0000", 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2440000000000004e-06 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286496+0000", 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016940182000000002 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.286550+0000", 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3640999999999997e-05 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.288163+0000", 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016126720000000001 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.452 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1734 4.f 4:f653a832:::smithi20438004-19:head [read 1042153~2571319,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.123178992000007, 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021098286000000001, 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.453 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1734 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269500+0000", 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269501+0000", 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2699999999999999e-07 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269504+0000", 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.58e-06 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269511+0000", 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6200000000000001e-06 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269517+0000", 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7030000000000003e-06 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269580+0000", 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3373000000000005e-05 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.269583+0000", 2024-09-13T11:34:03.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5020000000000001e-06 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289341+0000", 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019757429 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.289385+0000", 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3939000000000001e-05 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:46.290598+0000", 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012136130000000001 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.12.0:167 4.6 e51/31 4:6978b14d:::smithi20438004-23:head v 51'52, mlcod=45'50)", 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.201064649000003, 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013003298, 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.458 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191614+0000", 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.191616+0000", 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0270000000000001e-06 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196708+0000", 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050925720000000001 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196728+0000", 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9616999999999999e-05 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196738+0000", 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8330000000000006e-06 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196817+0000", 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9225000000000001e-05 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.196841+0000", 2024-09-13T11:34:03.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3972e-05 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204573+0000", 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077323629999999999 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.204618+0000", 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4688999999999999e-05 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1771 4.0 4:001d90c2:::smithi20438004-28:head [read 633380~1585144,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.122860027000002, 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013022562, 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1771 2024-09-13T11:34:03.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269819+0000", 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269820+0000", 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4000000000000002e-07 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269821+0000", 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9269999999999999e-06 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269825+0000", 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8789999999999996e-06 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269829+0000", 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4740000000000001e-06 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273144+0000", 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033155459999999999 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273155+0000", 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.062e-05 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282280+0000", 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091252620000000003 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282403+0000", 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012240499999999999 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282842+0000", 2024-09-13T11:34:03.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043900900000000002 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:1772 4.0 4:001d90c2:::smithi20438004-28:head [sparse-read 2038646~179878] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 76.122846581999994, 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013177762000000001, 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.468 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1772 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:34:03.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269833+0000", 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9499999999999998e-07 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269834+0000", 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2640000000000001e-06 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269837+0000", 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.005e-06 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.269843+0000", 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6389999999999997e-06 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273188+0000", 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033454840000000001 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.273196+0000", 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0830000000000004e-06 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282874+0000", 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096776810000000005 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.282924+0000", 2024-09-13T11:34:03.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0095000000000001e-05 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:47.283010+0000", 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6216000000000003e-05 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2324 4.16 4:6978b14d:::smithi20438004-23:head [write 544593~799729,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:34:03.473 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.255598601000003, 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.14427186, 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2324 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137081+0000", 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7100000000000001e-07 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137660+0000", 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00057879000000000003 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137671+0000", 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0518e-05 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137675+0000", 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8720000000000001e-06 2024-09-13T11:34:03.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137754+0000", 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8492000000000007e-05 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.137765+0000", 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1027e-05 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273433+0000", 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13566771999999999 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273569+0000", 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013616999999999999 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.273699+0000", 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000130142 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281227+0000", 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075284699999999998 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281259+0000", 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1610999999999999e-05 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281298+0000", 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9138999999999999e-05 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281352+0000", 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4138000000000002e-05 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2325 4.16 4:6978b14d:::smithi20438004-23:head [write 1761799~776065] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.248933285000007, 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13997515799999999, 2024-09-13T11:34:03.480 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2325 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143746+0000", 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.143748+0000", 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000001e-06 2024-09-13T11:34:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144385+0000", 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000637125 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144396+0000", 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0633e-05 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144401+0000", 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2469999999999999e-06 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144457+0000", 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5566999999999998e-05 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.144467+0000", 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8819999999999996e-06 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275371+0000", 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13090408000000001 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275440+0000", 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8749000000000002e-05 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.275530+0000", 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0236999999999997e-05 2024-09-13T11:34:03.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282215+0000", 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066849240000000001 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283575+0000", 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001360775 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283629+0000", 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3718999999999998e-05 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283721+0000", 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1799999999999995e-05 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2326 4.16 4:6978b14d:::smithi20438004-23:head [write 2948352~396241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:34:03.487 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.244567864000004, 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13583619799999999, 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2326 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148111+0000", 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148113+0000", 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1459999999999999e-06 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148305+0000", 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000192008 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148316+0000", 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0293e-05 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148327+0000", 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1489e-05 2024-09-13T11:34:03.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148398+0000", 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0421999999999998e-05 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148407+0000", 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1090000000000004e-06 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279356+0000", 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13094884700000001 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279444+0000", 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.279534+0000", 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9573000000000005e-05 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282581+0000", 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030473990000000001 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283856+0000", 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001275307 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283901+0000", 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5238999999999998e-05 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283947+0000", 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5972e-05 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2327 4.16 4:6978b14d:::smithi20438004-23:head [setxattr _header (56) in=7b,truncate 3344593] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.244340363999996, 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13580746099999999, 2024-09-13T11:34:03.494 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2327 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148339+0000", 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.34e-07 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148341+0000", 2024-09-13T11:34:03.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0999999999999998e-06 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148363+0000", 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2045999999999999e-05 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148369+0000", 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.665e-06 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148463+0000", 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4475999999999995e-05 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148474+0000", 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0319999999999999e-05 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280931+0000", 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132457142 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.280994+0000", 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2693000000000004e-05 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 6", 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281088+0000", 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3993000000000006e-05 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.282632+0000", 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001544199 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.283989+0000", 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013571670000000001 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284048+0000", 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9070999999999997e-05 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284146+0000", 2024-09-13T11:34:03.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8255e-05 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2328 4.16 4:6978b14d:::smithi20438004-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.244305849, 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.13592306100000001, 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.501 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2328 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148373+0000", 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148374+0000", 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6899999999999999e-07 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148375+0000", 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3599999999999999e-06 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148378+0000", 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0989999999999999e-06 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148381+0000", 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199999999999999e-06 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148500+0000", 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119052 2024-09-13T11:34:03.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.148509+0000", 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8759999999999994e-06 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281377+0000", 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.132868244 2024-09-13T11:34:03.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.281422+0000", 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4909e-05 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284171+0000", 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027497300000000001 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284216+0000", 2024-09-13T11:34:03.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4626999999999998e-05 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:56.284296+0000", 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0275e-05 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2687 4.10 4:0c9026a3:::smithi20438004-36:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:34:03.507 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 57.320853665000001, 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029783952999999998, 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2687 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:34:03.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071825+0000", 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071827+0000", 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.096e-06 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071831+0000", 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8210000000000001e-06 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071843+0000", 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1715e-05 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071848+0000", 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1959999999999999e-06 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071899+0000", 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0290000000000001e-05 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.071988+0000", 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9635000000000005e-05 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:06.101609+0000", 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0296212 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.511 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3509 4.f 4:f5aacf61:::smithi20438004-27:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.613334488, 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022491955000000001, 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.512 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3509 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779345+0000", 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779347+0000", 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9209999999999999e-06 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779354+0000", 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7980000000000006e-06 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779373+0000", 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289999999999999e-05 2024-09-13T11:34:03.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779399+0000", 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5945999999999999e-05 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779457+0000", 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8888000000000003e-05 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.779523+0000", 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5995999999999996e-05 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:23.801837+0000", 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022313116000000001 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4934 4.16 4:6d3e8486:::smithi20438004-2:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:58.584460+0000", 2024-09-13T11:34:03.516 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 4.8082191720000003, 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99117859500000005, 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4934 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584460+0000", 2024-09-13T11:34:03.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584460+0000", 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584460+0000", 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8000000000000002e-07 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584462+0000", 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.542e-06 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584466+0000", 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4259999999999996e-06 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584470+0000", 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2230000000000001e-06 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584524+0000", 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3168000000000003e-05 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:58.584618+0000", 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4745000000000002e-05 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:59.575639+0000", 2024-09-13T11:34:03.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99102021100000004 2024-09-13T11:34:03.521 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.521 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.521 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.521 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.521 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.521 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:34:03.522 INFO:tasks.rados.rados.0.smithi204.stdout:2044: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-13T11:34:03.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-13T11:34:03.523 INFO:tasks.ceph.osd.14.smithi154.stderr:2024-09-13T11:34:03.521+0000 7f7a77fee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 32889) UID: 0 2024-09-13T11:34:03.556 INFO:tasks.rados.rados.0.smithi204.stdout:seeking to 2328712 2024-09-13T11:34:03.565 INFO:tasks.rados.rados.0.smithi204.stdout:2043: finishing set_chunk to oid 25 2024-09-13T11:34:03.565 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 25 v 148 (ObjNum 439 snap 0 seq_num 439) dirty exists 2024-09-13T11:34:03.565 INFO:tasks.rados.rados.0.smithi204.stdout:2045: finishing write tid 1 to smithi20438004-4 2024-09-13T11:34:03.565 INFO:tasks.rados.rados.0.smithi204.stdout:2045: finishing write tid 2 to smithi20438004-4 2024-09-13T11:34:03.565 INFO:tasks.rados.rados.0.smithi204.stdout:2045: finishing write tid 3 to smithi20438004-4 2024-09-13T11:34:03.565 INFO:tasks.rados.rados.0.smithi204.stdout:2045: finishing write tid 4 to smithi20438004-4 2024-09-13T11:34:03.565 INFO:tasks.rados.rados.0.smithi204.stdout:2039: done (7 left) 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2041: done (6 left) 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2043: done (5 left) 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2044: done (4 left) 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 4 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2045: finishing write tid 5 to smithi20438004-4 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2045: oid 4 updating version 0 to 104 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2045: oid 4 version 104 is already newer than 102 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2045: oid 4 updating version 104 to 105 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2045: oid 4 version 105 is already newer than 103 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 4 v 105 (ObjNum 459 snap 0 seq_num 459) dirty exists 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2045: left oid 4 (ObjNum 459 snap 0 seq_num 459) 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2045: done (3 left) 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 3 2024-09-13T11:34:03.566 INFO:tasks.rados.rados.0.smithi204.stdout:2046: expect (ObjNum 453 snap 0 seq_num 453) 2024-09-13T11:34:03.624 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:34:03.623+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:34:03.668 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:34:03.668 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:34:03.668 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:34:03.668 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:34:03.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-13T11:34:03.724 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-09-13T11:34:03.723+0000 7f43ff28a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33163) UID: 0 2024-09-13T11:34:03.782 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:34:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:34:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-13T11:34:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-13T11:34:03.782 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:34:03.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-13T11:34:03.807 INFO:tasks.rados.rados.0.smithi204.stdout:2047: finishing write tid 1 to smithi20438004-1 2024-09-13T11:34:03.807 INFO:tasks.rados.rados.0.smithi204.stdout:2047: finishing write tid 2 to smithi20438004-1 2024-09-13T11:34:03.807 INFO:tasks.rados.rados.0.smithi204.stdout:2047: finishing write tid 3 to smithi20438004-1 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: finishing write tid 4 to smithi20438004-1 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2046: done (2 left) 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 2 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: finishing write tid 5 to smithi20438004-1 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: oid 1 updating version 0 to 124 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: oid 1 updating version 124 to 125 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: oid 1 version 125 is already newer than 123 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: oid 1 version 125 is already newer than 122 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 1 v 125 (ObjNum 460 snap 0 seq_num 460) dirty exists 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: left oid 1 (ObjNum 460 snap 0 seq_num 460) 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2047: done (1 left) 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout: waiting on 1 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2048: finishing write tid 1 to smithi20438004-22 2024-09-13T11:34:03.808 INFO:tasks.rados.rados.0.smithi204.stdout:2048: finishing write tid 2 to smithi20438004-22 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: finishing write tid 3 to smithi20438004-22 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: finishing write tid 4 to smithi20438004-22 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: finishing write tid 5 to smithi20438004-22 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: oid 22 updating version 0 to 130 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: oid 22 version 130 is already newer than 129 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: oid 22 updating version 130 to 131 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: oid 22 updating version 131 to 132 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:update_object_version oid 22 v 132 (ObjNum 461 snap 0 seq_num 461) dirty exists 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: left oid 22 (ObjNum 461 snap 0 seq_num 461) 2024-09-13T11:34:03.809 INFO:tasks.rados.rados.0.smithi204.stdout:2048: done (0 left) 2024-09-13T11:34:03.825 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:34:03.825+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:31 4.e e38/31 4:7022fc62:::smithi20438004-6:head v 38'22, mlcod=38'20)", 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:34:03.902 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 96.505275335999997, 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078075499999999999, 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394847+0000", 2024-09-13T11:34:03.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.394848+0000", 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4449999999999999e-06 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397492+0000", 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026441139999999999 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397513+0000", 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0381e-05 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397520+0000", 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4789999999999999e-06 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397586+0000", 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5687999999999999e-05 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.397612+0000", 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5870000000000001e-05 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402610+0000", 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049984199999999999 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:27.402654+0000", 2024-09-13T11:34:03.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4153000000000002e-05 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.15.0:47 4.e e44/31 4:7984cf84:::smithi20438004-46:head v 44'33, mlcod=43'31)", 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:34:03.907 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 86.868870243999993, 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081668139999999997, 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031252+0000", 2024-09-13T11:34:03.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.031253+0000", 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7900000000000007e-07 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037708+0000", 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064553029999999999 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037720+0000", 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2062000000000001e-05 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037745+0000", 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4766999999999999e-05 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037797+0000", 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2198999999999999e-05 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.037815+0000", 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8178000000000001e-05 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039386+0000", 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015708110000000001 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:32:37.039419+0000", 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2515000000000003e-05 2024-09-13T11:34:03.911 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2515 4.12 4:4d4c8512:::smithi20438004-29:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.822125681000003, 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008644934, 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.912 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2515 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077996+0000", 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.077997+0000", 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5999999999999991e-07 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078584+0000", 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00058694699999999999 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078595+0000", 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0808999999999999e-05 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078600+0000", 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9779999999999996e-06 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078667+0000", 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6445999999999994e-05 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078732+0000", 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5061000000000006e-05 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078820+0000", 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8208999999999995e-05 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080991+0000", 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021715419999999998 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084271+0000", 2024-09-13T11:34:03.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032798079999999999 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086533+0000", 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022614810000000001 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086566+0000", 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2959000000000001e-05 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086641+0000", 2024-09-13T11:34:03.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5734000000000004e-05 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2516 4.12 4:4d4c8512:::smithi20438004-29:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.821510973999999, 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082031369999999992, 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.918 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2516 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078611+0000", 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078612+0000", 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078631+0000", 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9823999999999999e-05 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078642+0000", 2024-09-13T11:34:03.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.011e-05 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078647+0000", 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4399999999999996e-06 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.079994+0000", 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001346835 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080073+0000", 2024-09-13T11:34:03.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8721999999999996e-05 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080232+0000", 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000159698 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.081037+0000", 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080509799999999997 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.084295+0000", 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032578360000000001 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086714+0000", 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024189799999999998 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086749+0000", 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4331e-05 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086814+0000", 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5703000000000004e-05 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2517 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.821467208999998, 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083158850000000003, 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.924 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2517 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:34:03.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078655+0000", 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9800000000000004e-07 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078658+0000", 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1940000000000001e-06 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078664+0000", 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1859999999999997e-06 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.078668+0000", 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6820000000000002e-06 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080476+0000", 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018074160000000001 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:03.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.080525+0000", 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9036000000000003e-05 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086839+0000", 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063138650000000001 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086884+0000", 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5717999999999998e-05 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.086971+0000", 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6290000000000002e-05 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:2537 4.12 4:4b088bf8:::smithi20438004-24:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.781171823999998, 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077763800000000003, 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.929 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2537 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118950+0000", 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118952+0000", 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0159999999999998e-06 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118976+0000", 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4227999999999999e-05 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.118987+0000", 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0475999999999999e-05 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119009+0000", 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2232999999999999e-05 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119058+0000", 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8427999999999998e-05 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.119146+0000", 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8393999999999994e-05 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:01.126727+0000", 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075806049999999998 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3033 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3663926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-13T11:34:03.933 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 49.401621968999997, 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083775140000000008, 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3033 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:34:03.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498500+0000", 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4499999999999998e-07 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498503+0000", 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4889999999999998e-06 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498508+0000", 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7079999999999996e-06 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.498511+0000", 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1879999999999998e-06 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503489+0000", 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049776849999999999 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.503558+0000", 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9128000000000006e-05 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:14.506878+0000", 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033199710000000001 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.937 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3042 4.12 4:4b088bf8:::smithi20438004-24:head [write 753013~645493] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 48.789780880000002, 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099953939999999995, 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3042 2024-09-13T11:34:03.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110341+0000", 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.110345+0000", 2024-09-13T11:34:03.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.563e-06 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111084+0000", 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073883200000000003 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111101+0000", 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7039999999999999e-05 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111110+0000", 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9060000000000008e-06 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111178+0000", 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8503000000000005e-05 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.111183+0000", 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2959999999999998e-06 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112893+0000", 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001709728 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.112975+0000", 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1757999999999999e-05 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.113113+0000", 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013859500000000001 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.119968+0000", 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068545869999999997 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120007+0000", 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8584000000000002e-05 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120085+0000", 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8226999999999995e-05 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120188+0000", 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103243 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:15.120337+0000", 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014853199999999999 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3369 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3663926,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:34:03.944 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 42.891023011000001, 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077891109999999996, 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3369 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009099+0000", 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009100+0000", 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1900000000000002e-07 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009105+0000", 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6899999999999997e-06 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009114+0000", 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1079999999999999e-06 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009121+0000", 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5320000000000003e-06 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.010369+0000", 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001247915 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.010466+0000", 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7579999999999997e-05 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.016888+0000", 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064216669999999998 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.948 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3370 4.12 4:4d4c8512:::smithi20438004-29:head [read 32972~3630954,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 42.890989783999999, 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010949787000000001, 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.949 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3370 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009132+0000", 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009133+0000", 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7299999999999995e-07 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009136+0000", 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5090000000000002e-06 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009144+0000", 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2880000000000004e-06 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.009150+0000", 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.016932+0000", 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077818109999999996 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.017048+0000", 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011644799999999999 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:21.020082+0000", 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030339849999999999 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3846 4.12 4:4ba9f189:::smithi20438004-14:head [write 497573~628943,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.587147+0000", 2024-09-13T11:34:03.953 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.312974648000001, 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010066912000000001, 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3846 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.587147+0000", 2024-09-13T11:34:03.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.587147+0000", 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.587150+0000", 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7350000000000001e-06 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.588997+0000", 2024-09-13T11:34:03.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001846952 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589014+0000", 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7011e-05 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589024+0000", 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8859999999999999e-06 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589103+0000", 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8969999999999998e-05 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589171+0000", 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7947999999999994e-05 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,8", 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.589277+0000", 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105714 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.593005+0000", 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037278720000000001 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.593997+0000", 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00099221599999999997 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.597083+0000", 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003086063 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.597130+0000", 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7005999999999997e-05 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.597214+0000", 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4538999999999998e-05 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.959 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3868 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~3546310,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.781476+0000", 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.118646462000001, 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075397750000000003, 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3868 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781476+0000", 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781476+0000", 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781477+0000", 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6020000000000001e-06 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781488+0000", 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0608000000000001e-05 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781503+0000", 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4932e-05 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781510+0000", 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3010000000000001e-06 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781577+0000", 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7436999999999998e-05 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781690+0000", 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011268399999999999 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.789015+0000", 2024-09-13T11:34:03.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073252109999999999 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3869 4.12 4:4b088bf8:::smithi20438004-24:head [read 2720685~825625,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.781527+0000", 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.118594991999998, 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085861070000000008, 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.964 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3869 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781527+0000", 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781527+0000", 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781528+0000", 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3799999999999997e-07 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781533+0000", 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6200000000000004e-06 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781539+0000", 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3229999999999997e-06 2024-09-13T11:34:03.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781542+0000", 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3909999999999998e-06 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.789057+0000", 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075145059999999998 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.789122+0000", 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5436000000000002e-05 2024-09-13T11:34:03.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.790113+0000", 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00099119300000000002 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:3870 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~3546310,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 33.118557152999998, 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012108255, 2024-09-13T11:34:03.968 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3870 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781565+0000", 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0999999999999999e-07 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781568+0000", 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.424e-06 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781572+0000", 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6800000000000001e-06 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.781576+0000", 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9899999999999999e-06 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.790135+0000", 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085584449999999996 2024-09-13T11:34:03.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.790187+0000", 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1683e-05 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:30.793673+0000", 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034866229999999999 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.972 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4151 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3411854,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.005478041, 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093747399999999995, 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4151 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894644+0000", 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9400000000000001e-07 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894647+0000", 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0369999999999999e-06 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894654+0000", 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9120000000000001e-06 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894659+0000", 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5600000000000004e-06 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.897162+0000", 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025029829999999999 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.897254+0000", 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1594000000000004e-05 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.904019+0000", 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067652600000000004 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.976 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4152 4.12 4:4d4c8512:::smithi20438004-29:head [sparse-read 0~2953102] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:38.894681+0000", 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.005441033, 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011384893, 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.977 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4152 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894681+0000", 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894681+0000", 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894682+0000", 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9399999999999995e-07 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894684+0000", 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4080000000000001e-06 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894689+0000", 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2129999999999999e-06 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.894693+0000", 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2050000000000004e-06 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.904055+0000", 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093621360000000001 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.904118+0000", 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2226999999999999e-05 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:38.906066+0000", 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019482099999999999 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4256 4.12 4:4d4c8512:::smithi20438004-29:head [read 0~3411854,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:40.355909+0000", 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.544213306, 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083641099999999993, 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.981 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4256 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355909+0000", 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355909+0000", 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355910+0000", 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6000000000000003e-07 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355917+0000", 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8900000000000007e-06 2024-09-13T11:34:03.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355926+0000", 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5359999999999992e-06 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.355950+0000", 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3813000000000001e-05 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.359612+0000", 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036622780000000001 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.359678+0000", 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6050999999999995e-05 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:40.364273+0000", 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045947820000000004 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4531 4.12 4:4b088bf8:::smithi20438004-24:head [sparse-read 0~944644,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:49.588607+0000", 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.311515338, 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075317179999999997, 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4531 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.986 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588607+0000", 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588607+0000", 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588607+0000", 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6899999999999999e-07 2024-09-13T11:34:03.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588614+0000", 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2110000000000001e-06 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588618+0000", 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0280000000000002e-06 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588621+0000", 2024-09-13T11:34:03.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.88e-06 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.595025+0000", 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064043640000000001 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.595079+0000", 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3579999999999999e-05 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.596138+0000", 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010593860000000001 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4532 4.12 4:4b088bf8:::smithi20438004-24:head [read 1149655~951286,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:49.588625+0000", 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.311497103000001, 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080854059999999998, 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.990 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4532 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588625+0000", 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588625+0000", 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588625+0000", 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7300000000000002e-07 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588627+0000", 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.61e-06 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588631+0000", 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9090000000000002e-06 2024-09-13T11:34:03.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588634+0000", 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7070000000000001e-06 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.596167+0000", 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075334829999999997 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.596220+0000", 2024-09-13T11:34:03.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2528999999999998e-05 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.596710+0000", 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00049079500000000003 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.4631.0:4533 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~2100941,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-13T11:33:49.588637+0000", 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.311484882, 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088776770000000005, 2024-09-13T11:34:03.994 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.4631", 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4533 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588637+0000", 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588637+0000", 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588637+0000", 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4900000000000002e-07 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588639+0000", 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5829999999999999e-06 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588643+0000", 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8079999999999998e-06 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.588645+0000", 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5569999999999998e-06 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.596731+0000", 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080859169999999998 2024-09-13T11:34:03.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.596775+0000", 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3513999999999997e-05 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-13T11:33:49.597515+0000", 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000740049 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-13T11:34:03.998 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:34:04.000 INFO:tasks.rados.rados.0.smithi204.stderr:0 errors. 2024-09-13T11:34:04.001 INFO:tasks.rados.rados.0.smithi204.stderr: 2024-09-13T11:34:04.001 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-13T11:34:04.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:34:04.002 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:34:03.998+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:34:04.101 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:34:04.100+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:34:04.144 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:04.144 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:04.144 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:34:04.145 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:04.153 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-13T11:34:04.202 INFO:tasks.ceph.osd.11.smithi204.stderr:2024-09-13T11:34:04.201+0000 7f955c4d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33341) UID: 0 2024-09-13T11:34:04.255 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:04.255 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:34:04.256 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:34:04.256 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:04.264 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-13T11:34:04.304 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-13T11:34:04.302+0000 7f3f80a5a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 32873) UID: 0 2024-09-13T11:34:04.311 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:34:04.311 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":102,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:34:02.569717+0000","last_up_change":"2024-09-13T11:32:56.264138+0000","last_in_change":"2024-09-13T11:33:57.535495+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:32:02.947125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.97729301452636719,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:32:05.120118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":0.97729301452636719,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T11:32:17.683275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":7.9499998092651367,"score_stable":7.9499998092651367,"optimal_score":0.43999999761581421,"raw_score_acting":3.5,"raw_score_stable":3.5,"primary_affinity_weighted":0.97729301452636719,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-09-13T11:32:18.790550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":36,"pg_placement_num":36,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":36,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"87","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.68999999761581421,"raw_score_acting":1.8300000429153442,"raw_score_stable":1.8300000429153442,"primary_affinity_weighted":0.97729301452636719,"average_primary_affinity":0.98000001907348633,"average_primary_affinity_weighted":0.98000001907348633}}],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2677506544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2677506544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2677506544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":2677506544}]},"public_addr":"172.21.15.38:6805/2677506544","cluster_addr":"172.21.15.38:6806/2677506544","heartbeat_back_addr":"172.21.15.38:6808/2677506544","heartbeat_front_addr":"172.21.15.38:6807/2677506544","state":["exists","up"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":3561806747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":3561806747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":3561806747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":3561806747}]},"public_addr":"172.21.15.118:6808/3561806747","cluster_addr":"172.21.15.118:6809/3561806747","heartbeat_back_addr":"172.21.15.118:6811/3561806747","heartbeat_front_addr":"172.21.15.118:6810/3561806747","state":["exists","up"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":41,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4217282489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4217282489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4217282489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4217282489}]},"public_addr":"172.21.15.154:6800/4217282489","cluster_addr":"172.21.15.154:6801/4217282489","heartbeat_back_addr":"172.21.15.154:6803/4217282489","heartbeat_front_addr":"172.21.15.154:6802/4217282489","state":["exists"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":1,"in":1,"weight":1,"primary_affinity":0.636688232421875,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2326505959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":2326505959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":2326505959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2326505959}]},"public_addr":"172.21.15.204:6804/2326505959","cluster_addr":"172.21.15.204:6805/2326505959","heartbeat_back_addr":"172.21.15.204:6807/2326505959","heartbeat_front_addr":"172.21.15.204:6806/2326505959","state":["exists","up"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":3107658120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3107658120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3107658120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":3107658120}]},"public_addr":"172.21.15.38:6801/3107658120","cluster_addr":"172.21.15.38:6802/3107658120","heartbeat_back_addr":"172.21.15.38:6804/3107658120","heartbeat_front_addr":"172.21.15.38:6803/3107658120","state":["exists","up"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3800782738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":3800782738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":3800782738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":3800782738}]},"public_addr":"172.21.15.118:6800/3800782738","cluster_addr":"172.21.15.118:6801/3800782738","heartbeat_back_addr":"172.21.15.118:6803/3800782738","heartbeat_front_addr":"172.21.15.118:6802/3800782738","state":["exists","up"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3986362716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3986362716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3986362716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3986362716}]},"public_addr":"172.21.15.154:6808/3986362716","cluster_addr":"172.21.15.154:6809/3986362716","heartbeat_back_addr":"172.21.15.154:6811/3986362716","heartbeat_front_addr":"172.21.15.154:6810/3986362716","state":["exists","up"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1978982202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":1978982202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":1978982202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1978982202}]},"public_addr":"172.21.15.204:6808/1978982202","cluster_addr":"172.21.15.204:6809/1978982202","heartbeat_back_addr":"172.21.15.204:6811/1978982202","heartbeat_front_addr":"172.21.15.204:6810/1978982202","state":["exists","up"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3608171775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3608171775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3608171775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3608171775}]},"public_addr":"172.21.15.38:6813/3608171775","cluster_addr":"172.21.15.38:6814/3608171775","heartbeat_back_addr":"172.21.15.38:6816/3608171775","heartbeat_front_addr":"172.21.15.38:6815/3608171775","state":["exists","up"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2580659844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2580659844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":2580659844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2580659844}]},"public_addr":"172.21.15.118:6804/2580659844","cluster_addr":"172.21.15.118:6805/2580659844","heartbeat_back_addr":"172.21.15.118:6807/2580659844","heartbeat_front_addr":"172.21.15.118:6806/2580659844","state":["exists","up"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2352891695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":2352891695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":2352891695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2352891695}]},"public_addr":"172.21.15.154:6812/2352891695","cluster_addr":"172.21.15.154:6813/2352891695","heartbeat_back_addr":"172.21.15.154:6815/2352891695","heartbeat_front_addr":"172.21.15.154:6814/2352891695","state":["exists","up"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2304836889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":2304836889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":2304836889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2304836889}]},"public_addr":"172.21.15.204:6800/2304836889","cluster_addr":"172.21.15.204:6801/2304836889","heartbeat_back_addr":"172.21.15.204:6803/2304836889","heartbeat_front_addr":"172.21.15.204:6802/2304836889","state":["exists","up"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":58,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":1688419254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":1688419254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":1688419254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":1688419254}]},"public_addr":"172.21.15.38:6809/1688419254","cluster_addr":"172.21.15.38:6810/1688419254","heartbeat_back_addr":"172.21.15.38:6812/1688419254","heartbeat_front_addr":"172.21.15.38:6811/1688419254","state":["exists"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":3727425991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":3727425991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":3727425991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":3727425991}]},"public_addr":"172.21.15.118:6812/3727425991","cluster_addr":"172.21.15.118:6813/3727425991","heartbeat_back_addr":"172.21.15.118:6815/3727425991","heartbeat_front_addr":"172.21.15.118:6814/3727425991","state":["exists","up"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":538351706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":538351706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":538351706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":538351706}]},"public_addr":"172.21.15.154:6804/538351706","cluster_addr":"172.21.15.154:6805/538351706","heartbeat_back_addr":"172.21.15.154:6807/538351706","heartbeat_front_addr":"172.21.15.154:6806/538351706","state":["exists","up"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":501019151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":501019151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":501019151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":501019151}]},"public_addr":"172.21.15.204:6812/501019151","cluster_addr":"172.21.15.204:6813/501019151","heartbeat_back_addr":"172.21.15.204:6815/501019151","heartbeat_front_addr":"172.21.15.204:6814/501019151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:03.855267+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-13T11:33:26.046732+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:00.198442+0000","dead_epoch":41},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:04.310901+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:03.659118+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-13T11:33:09.623455+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:19.808086+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:19.499908+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:07.828658+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-13T11:33:22.980747+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:27.377283+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:26.397265+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:32:01.027609+0000","dead_epoch":59},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:19.963999+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:19.534738+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:33:02.472566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.7","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.2","osds":[13,3,15]},{"pgid":"3.3","osds":[13,7,15]},{"pgid":"4.2","osds":[3,12,9]},{"pgid":"4.4","osds":[3,8,5]},{"pgid":"4.11","osds":[7,3,15]},{"pgid":"4.12","osds":[14,8,3]},{"pgid":"4.13","osds":[7,13,14]},{"pgid":"4.14","osds":[4,7,3]},{"pgid":"4.16","osds":[10,6,12]},{"pgid":"4.19","osds":[6,0,7]},{"pgid":"4.1b","osds":[0,11,4]},{"pgid":"4.20","osds":[7,8,10]},{"pgid":"4.22","osds":[15,6,3]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:34:04.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:34:04.404 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:34:04.403+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:34:04.410 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:34:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:34:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:34:04.410 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4344.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 118.053600479, 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.40010368499999999, 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4344", 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.38:0/366635618", 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.411 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354312+0000", 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354319+0000", 2024-09-13T11:34:04.412 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8549999999999996e-06 2024-09-13T11:34:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354324+0000", 2024-09-13T11:34:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1869999999999996e-06 2024-09-13T11:34:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.413 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354331+0000", 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4139999999999997e-06 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354337+0000", 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000004e-06 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354383+0000", 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6042999999999999e-05 2024-09-13T11:34:04.414 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.354393+0000", 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2979999999999997e-06 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754331+0000", 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.39993873000000002 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:06.754416+0000", 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4448000000000002e-05 2024-09-13T11:34:04.415 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:208 3.2 e38/30 3:6c91e224:::smithi20438004-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'2, mlcod=0'0)", 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.305069558, 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030901084999999998, 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.416 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102843+0000", 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.102844+0000", 2024-09-13T11:34:04.417 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9400000000000005e-07 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115010+0000", 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012166273999999999 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115026+0000", 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5880999999999999e-05 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.418 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115035+0000", 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5060000000000001e-06 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115099+0000", 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3782999999999996e-05 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115129+0000", 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0046e-05 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.419 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133700+0000", 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018570606 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133744+0000", 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4394999999999998e-05 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.420 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:216 3.2 e38/30 3:696650e2:::smithi20438004-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.292851963999993, 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030932256000000002, 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.421 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115061+0000", 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7700000000000003e-07 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121738+0000", 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0066762649999999998 2024-09-13T11:34:04.422 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121758+0000", 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0041999999999999e-05 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121779+0000", 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0885000000000001e-05 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.423 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121885+0000", 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106119 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121912+0000", 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7701000000000001e-05 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.424 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145947+0000", 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024034436999999999 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145993+0000", 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5930000000000002e-05 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1679 4.4 4:2b094039:::smithi20438004-40:head [tier-flush] snapc 0=[] RETRY=3 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e49)", 2024-09-13T11:34:04.425 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 79.138727857000006, 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0158098280000001, 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1679 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.426 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269185+0000", 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269187+0000", 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9479999999999998e-06 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.427 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269193+0000", 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.173e-06 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269209+0000", 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6039e-05 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.428 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.269224+0000", 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5494000000000001e-05 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270767+0000", 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0015428869999999999 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.429 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.270782+0000", 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.519e-05 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284573+0000", 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013791077000000001 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:45.284717+0000", 2024-09-13T11:34:04.430 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014369599999999999 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284994+0000", 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.000277324 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2310 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:04.431 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.308106312000007, 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.17044168200000001, 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2310 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.432 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099806+0000", 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.099808+0000", 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8819999999999999e-06 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.433 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100126+0000", 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00031826000000000001 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100144+0000", 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8029999999999998e-05 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100153+0000", 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0119999999999994e-06 2024-09-13T11:34:04.434 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100230+0000", 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6221000000000001e-05 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.100242+0000", 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.257e-05 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270222+0000", 2024-09-13T11:34:04.435 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16997934100000001 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270248+0000", 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6366000000000001e-05 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2311 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.301566035999997, 2024-09-13T11:34:04.436 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16403588, 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2311 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.437 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106346+0000", 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106347+0000", 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7899999999999997e-07 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106673+0000", 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588799999999998 2024-09-13T11:34:04.438 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106687+0000", 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3247e-05 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106693+0000", 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2249999999999997e-06 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106741+0000", 2024-09-13T11:34:04.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8715999999999999e-05 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.106750+0000", 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3159999999999997e-06 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270375+0000", 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.16362501300000001 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270382+0000", 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5959999999999998e-06 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2312 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:34:04.441 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.296274773999997, 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158778577, 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2312 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111638+0000", 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111639+0000", 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.567e-06 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111756+0000", 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000117166 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111770+0000", 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3492999999999999e-05 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111778+0000", 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5820000000000004e-06 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111834+0000", 2024-09-13T11:34:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6564000000000003e-05 2024-09-13T11:34:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111842+0000", 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8280000000000003e-06 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270410+0000", 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15856827400000001 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270416+0000", 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1029999999999998e-06 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2313 4.18 4.8a282f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.296120443000007, 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158679564, 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:04.447 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2313 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:34:04.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111792+0000", 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6399999999999998e-07 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111795+0000", 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0450000000000001e-06 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111799+0000", 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.5250000000000001e-06 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111802+0000", 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.162e-06 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111876+0000", 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4062999999999999e-05 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111885+0000", 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0850000000000003e-06 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270465+0000", 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158579839 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270472+0000", 2024-09-13T11:34:04.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4810000000000003e-06 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4.8a282f38 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.296090226999993, 2024-09-13T11:34:04.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.15868154300000001, 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111822+0000", 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111823+0000", 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3299999999999998e-07 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111824+0000", 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7290000000000001e-06 2024-09-13T11:34:04.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111829+0000", 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5859999999999998e-06 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111832+0000", 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.456e-06 2024-09-13T11:34:04.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111901+0000", 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8447000000000004e-05 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.111905+0000", 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8569999999999997e-06 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270497+0000", 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.158592381 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.270504+0000", 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.754e-06 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.457 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2309 4.14 4:2c957d44:::smithi20438004-22:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e59)", 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 68.138595108999993, 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.00703426, 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2309 2024-09-13T11:34:04.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269317+0000", 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269319+0000", 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.265e-06 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269324+0000", 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1800000000000004e-06 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269334+0000", 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7310000000000005e-06 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269343+0000", 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8339999999999994e-06 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269381+0000", 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7709000000000001e-05 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.269391+0000", 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5599999999999999e-06 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278002+0000", 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086114859999999998 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.278150+0000", 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014796199999999999 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.276352+0000", 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.99820153300000003 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2170 4.12 4:4b088bf8:::smithi20438004-24:head [write 2083958~603055] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.130029246999996, 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029083859, 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2170 2024-09-13T11:34:04.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277883+0000", 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.277884+0000", 2024-09-13T11:34:04.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4399999999999998e-07 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278124+0000", 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000239485 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278133+0000", 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3219999999999999e-06 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278141+0000", 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280144+0000", 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0020033289999999999 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280155+0000", 2024-09-13T11:34:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0966e-05 2024-09-13T11:34:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301300+0000", 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021145557999999998 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301414+0000", 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011306099999999999 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.301601+0000", 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000187357 2024-09-13T11:34:04.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304518+0000", 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029169539999999998 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304561+0000", 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3229000000000002e-05 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306715+0000", 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002153706 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306792+0000", 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7466999999999995e-05 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306851+0000", 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8811999999999999e-05 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306967+0000", 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000116062 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2171 4.12 4:4b088bf8:::smithi20438004-24:head [setxattr _header (56) in=7b,truncate 3058874] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.129756678999996, 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029067939000000001, 2024-09-13T11:34:04.472 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2171 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278156+0000", 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278157+0000", 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0699999999999996e-07 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278161+0000", 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2640000000000002e-06 2024-09-13T11:34:04.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278173+0000", 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1868999999999999e-05 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278179+0000", 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280537+0000", 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023576320000000001 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280544+0000", 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7560000000000002e-06 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303651+0000", 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023107240000000001 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303802+0000", 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000150774 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 8,9,14", 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303982+0000", 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018009800000000001 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306699+0000", 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027165100000000001 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306720+0000", 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1535e-05 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.306753+0000", 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3019999999999999e-05 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307010+0000", 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025724800000000001 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307061+0000", 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0633000000000001e-05 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307224+0000", 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000162843 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.480 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2172 4.12 4:4b088bf8:::smithi20438004-24:head [read 0~1] snapc 0=[] RETRY=3 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.129723995000006, 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029398245999999999, 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2172 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278189+0000", 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6300000000000005e-07 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278193+0000", 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2899999999999996e-06 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278201+0000", 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.221e-06 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278207+0000", 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4289999999999998e-06 2024-09-13T11:34:04.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280568+0000", 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023610900000000002 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280577+0000", 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8259999999999999e-06 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304605+0000", 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.024027829000000001 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.304704+0000", 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9610000000000003e-05 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307334+0000", 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002629667 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307421+0000", 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7433000000000001e-05 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307587+0000", 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016528800000000001 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2219 4.12 4:4d4c8512:::smithi20438004-29:head [sparse-read 0~2348017] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.129652786999998, 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029874108999999999, 2024-09-13T11:34:04.487 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2219 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278260+0000", 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3599999999999999e-07 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278262+0000", 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2110000000000001e-06 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278269+0000", 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.314e-06 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.278274+0000", 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5249999999999996e-06 2024-09-13T11:34:04.490 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280850+0000", 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025761389999999999 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280858+0000", 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8760000000000005e-06 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.491 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307628+0000", 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.026769465999999999 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307703+0000", 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4847999999999999e-05 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.308134+0000", 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00043129399999999998 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.492 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2314 4.18 4:1cf41451:::smithi20438004-7:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.127897320000002, 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029229465, 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.493 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2314 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280015+0000", 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280016+0000", 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8000000000000005e-07 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280020+0000", 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8500000000000004e-06 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280026+0000", 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1070000000000001e-06 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280032+0000", 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3289999999999999e-06 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280109+0000", 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6298999999999996e-05 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280116+0000", 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5129999999999999e-06 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298271+0000", 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018154805 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298334+0000", 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3268999999999993e-05 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309030+0000", 2024-09-13T11:34:04.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010695566 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309089+0000", 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9630000000000003e-05 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309245+0000", 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000155317 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2340 4.18 4:1d367c64:::smithi20438004-45:head [write 578207~550227] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.127869572999998, 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.030475704999999999, 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2340 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.499 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280043+0000", 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0399999999999996e-07 2024-09-13T11:34:04.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280452+0000", 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00040812800000000001 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280461+0000", 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3829999999999998e-06 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280468+0000", 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0400000000000004e-06 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280506+0000", 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8037000000000002e-05 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280514+0000", 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.011e-06 2024-09-13T11:34:04.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298673+0000", 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018159411 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.298829+0000", 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015594 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:34:04.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.299023+0000", 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193838 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307549+0000", 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0085256289999999998 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307635+0000", 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6596999999999998e-05 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307689+0000", 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4027999999999998e-05 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307748+0000", 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8733999999999997e-05 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310255+0000", 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025070940000000001 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310375+0000", 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000119691 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310519+0000", 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014364 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2341 4.18 4:1d367c64:::smithi20438004-45:head [write 1762410~779734] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.127427205999993, 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031687067999999999, 2024-09-13T11:34:04.507 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2341 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280485+0000", 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.280486+0000", 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1099999999999995e-07 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281018+0000", 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053231899999999996 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281027+0000", 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.704e-06 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281034+0000", 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0659999999999998e-06 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281159+0000", 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000124661 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281168+0000", 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8400000000000001e-06 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302725+0000", 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021557478000000001 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.302889+0000", 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00016421299999999999 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.303087+0000", 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00019817200000000001 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307806+0000", 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.004718168 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307868+0000", 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2775999999999997e-05 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307920+0000", 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1749000000000002e-05 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.307971+0000", 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1343e-05 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311829+0000", 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0038571769999999998 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312043+0000", 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021412199999999999 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312172+0000", 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012966900000000001 2024-09-13T11:34:04.514 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2342 4.18 4:1d367c64:::smithi20438004-45:head [write 3286845~91362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.126860054999995, 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031505618999999999, 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.515 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2342 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281052+0000", 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281053+0000", 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2799999999999995e-07 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281104+0000", 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0550999999999998e-05 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281112+0000", 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0129999999999993e-06 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281118+0000", 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2049999999999999e-06 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281191+0000", 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2738999999999996e-05 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281197+0000", 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5729999999999996e-06 2024-09-13T11:34:04.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309351+0000", 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028153293999999999 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309441+0000", 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0048000000000002e-05 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.309594+0000", 2024-09-13T11:34:04.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000153609 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311315+0000", 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0017209829999999999 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311795+0000", 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048009300000000001 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312281+0000", 2024-09-13T11:34:04.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00048612699999999999 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312394+0000", 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000112935 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312404+0000", 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0112000000000001e-05 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312449+0000", 2024-09-13T11:34:04.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4650000000000001e-05 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312558+0000", 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108959 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2343 4.18 4:1d367c64:::smithi20438004-45:head [setxattr _header (56) in=7b,truncate 3378207] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.126784505000003, 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031723945000000003, 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.522 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2343 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281128+0000", 2024-09-13T11:34:04.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281129+0000", 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281132+0000", 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.988e-06 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281137+0000", 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.95e-06 2024-09-13T11:34:04.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281142+0000", 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4e-06 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281219+0000", 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6946000000000005e-05 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281225+0000", 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.601e-06 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310552+0000", 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.029326497 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310643+0000", 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1416999999999996e-05 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4,6,11", 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.310817+0000", 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000173463 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312229+0000", 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001412424 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312324+0000", 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.4264000000000004e-05 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312578+0000", 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00025401699999999999 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312605+0000", 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.7557000000000001e-05 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312650+0000", 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4496000000000002e-05 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312683+0000", 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2898000000000002e-05 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312852+0000", 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000169436 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2344 4.18 4:1d367c64:::smithi20438004-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.126764757999993, 2024-09-13T11:34:04.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.032021714999999999, 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2344 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281148+0000", 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.15e-07 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281150+0000", 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.857e-06 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281155+0000", 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0579999999999997e-06 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281160+0000", 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.0409999999999997e-06 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281248+0000", 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7979000000000006e-05 2024-09-13T11:34:04.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.281255+0000", 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5089999999999999e-06 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311169+0000", 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.02991394 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for rw locks", 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.311266+0000", 2024-09-13T11:34:04.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7007000000000001e-05 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312894+0000", 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0016280489999999999 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.312976+0000", 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2052999999999995e-05 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.313169+0000", 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000193807 2024-09-13T11:34:04.534 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.535 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.535 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:04.537 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-13T11:34:04.537 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:34:04.535+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:34:04.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-13T11:34:04.637 INFO:tasks.ceph.osd.8.smithi038.stderr:2024-09-13T11:34:04.636+0000 7f36a729b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33165) UID: 0 2024-09-13T11:34:04.663 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:34:04.664 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:34:04.675 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-13T11:34:02.952628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":242960363,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":3980,"num_read_kb":2412963,"num_write":2864,"num_write_kb":630094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":105781001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":125,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":965980,"kb_used_data":573068,"kb_used_omap":20,"kb_used_meta":392683,"kb_avail":1225867940,"statfs":{"total":1256277934080,"available":1255288770560,"internally_reserved":0,"allocated":586821632,"data_stored":581237391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20732,"internal_metadata":402108164},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22981099,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":-41,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":310,"num_read_kb":230434,"num_write":214,"num_write_kb":49923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":10430429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000999"},"pg_stats":[{"pgid":"4.23","version":"101'76","reported_seq":340,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:02.239756+0000","last_change":"2024-09-13T11:33:51.879763+0000","last_active":"2024-09-13T11:34:02.239756+0000","last_peered":"2024-09-13T11:34:02.239756+0000","last_clean":"2024-09-13T11:34:02.239756+0000","last_became_active":"2024-09-13T11:33:51.297481+0000","last_became_peered":"2024-09-13T11:33:51.297481+0000","last_unstale":"2024-09-13T11:34:02.239756+0000","last_undegraded":"2024-09-13T11:34:02.239756+0000","last_fullsized":"2024-09-13T11:34:02.239756+0000","mapping_epoch":92,"log_start":"101'73","ondisk_log_start":"101'73","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":6,"last_scrub":"86'69","last_scrub_stamp":"2024-09-13T11:33:51.879699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:51.879699+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":73,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:15.956591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6168301,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":45087,"num_write":6,"num_write_kb":2952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,15],"acting":[14,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"82'143","reported_seq":476,"reported_epoch":98,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:58.547636+0000","last_change":"2024-09-13T11:33:51.008226+0000","last_active":"2024-09-13T11:33:58.547636+0000","last_peered":"2024-09-13T11:33:58.547636+0000","last_clean":"2024-09-13T11:33:58.547636+0000","last_became_active":"2024-09-13T11:33:50.287139+0000","last_became_peered":"2024-09-13T11:33:50.287139+0000","last_unstale":"2024-09-13T11:33:58.547636+0000","last_undegraded":"2024-09-13T11:33:58.547636+0000","last_fullsized":"2024-09-13T11:33:58.547636+0000","mapping_epoch":91,"log_start":"81'140","ondisk_log_start":"81'140","created":87,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":6,"last_scrub":"82'143","last_scrub_stamp":"2024-09-13T11:33:51.008129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:51.008129+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":137,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:12.154762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6],"acting":[15,6,3],"avail_no_missing":["15","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.21","version":"95'138","reported_seq":542,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.311525+0000","last_change":"2024-09-13T11:34:01.303990+0000","last_active":"2024-09-13T11:34:01.311525+0000","last_peered":"2024-09-13T11:34:01.311525+0000","last_clean":"2024-09-13T11:34:01.311525+0000","last_became_active":"2024-09-13T11:34:01.292050+0000","last_became_peered":"2024-09-13T11:34:01.292050+0000","last_unstale":"2024-09-13T11:34:01.311525+0000","last_undegraded":"2024-09-13T11:34:01.311525+0000","last_fullsized":"2024-09-13T11:34:01.311525+0000","mapping_epoch":100,"log_start":"95'135","ondisk_log_start":"95'135","created":87,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"85'126","last_scrub_stamp":"2024-09-13T11:33:49.967915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:49.967915+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":133,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:07.735341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5714512,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":37592,"num_write":12,"num_write_kb":3046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4921621,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,10,13],"acting":[14,10,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.20","version":"93'94","reported_seq":386,"reported_epoch":98,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:58.551937+0000","last_change":"2024-09-13T11:33:50.414891+0000","last_active":"2024-09-13T11:33:58.551937+0000","last_peered":"2024-09-13T11:33:58.551937+0000","last_clean":"2024-09-13T11:33:58.551937+0000","last_became_active":"2024-09-13T11:33:49.495571+0000","last_became_peered":"2024-09-13T11:33:49.495571+0000","last_unstale":"2024-09-13T11:33:58.551937+0000","last_undegraded":"2024-09-13T11:33:58.551937+0000","last_fullsized":"2024-09-13T11:33:58.551937+0000","mapping_epoch":90,"log_start":"91'91","ondisk_log_start":"91'91","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":6,"last_scrub":"91'91","last_scrub_stamp":"2024-09-13T11:33:50.414786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:50.414786+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:02.432532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823642,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7,8,10],"avail_no_missing":["7","8","10"],"object_location_counts":[{"shards":"7,8,10","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"91'137","reported_seq":534,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.285804+0000","last_change":"2024-09-13T11:33:51.922263+0000","last_active":"2024-09-13T11:34:01.285804+0000","last_peered":"2024-09-13T11:34:01.285804+0000","last_clean":"2024-09-13T11:34:01.285804+0000","last_became_active":"2024-09-13T11:33:51.295367+0000","last_became_peered":"2024-09-13T11:33:51.295367+0000","last_unstale":"2024-09-13T11:34:01.285804+0000","last_undegraded":"2024-09-13T11:34:01.285804+0000","last_fullsized":"2024-09-13T11:34:01.285804+0000","mapping_epoch":92,"log_start":"85'131","ondisk_log_start":"85'131","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'137","last_scrub_stamp":"2024-09-13T11:33:51.922195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:51.922195+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":131,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:52.055981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756414,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":28574,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,11],"acting":[0,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"86'106","reported_seq":473,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.286076+0000","last_change":"2024-09-13T11:33:51.296282+0000","last_active":"2024-09-13T11:34:01.286076+0000","last_peered":"2024-09-13T11:34:01.286076+0000","last_clean":"2024-09-13T11:34:01.286076+0000","last_became_active":"2024-09-13T11:33:51.295794+0000","last_became_peered":"2024-09-13T11:33:51.295794+0000","last_unstale":"2024-09-13T11:34:01.286076+0000","last_undegraded":"2024-09-13T11:34:01.286076+0000","last_fullsized":"2024-09-13T11:34:01.286076+0000","mapping_epoch":92,"log_start":"85'103","ondisk_log_start":"85'103","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'106","last_scrub_stamp":"2024-09-13T11:33:49.899179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:49.899179+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":103,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:59.648281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2470695,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6],"acting":[14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1d","version":"101'114","reported_seq":446,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.285824+0000","last_change":"2024-09-13T11:33:51.294606+0000","last_active":"2024-09-13T11:34:01.285824+0000","last_peered":"2024-09-13T11:34:01.285824+0000","last_clean":"2024-09-13T11:34:01.285824+0000","last_became_active":"2024-09-13T11:33:51.294185+0000","last_became_peered":"2024-09-13T11:33:51.294185+0000","last_unstale":"2024-09-13T11:34:01.285824+0000","last_undegraded":"2024-09-13T11:34:01.285824+0000","last_fullsized":"2024-09-13T11:34:01.285824+0000","mapping_epoch":92,"log_start":"94'111","ondisk_log_start":"94'111","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":5,"last_scrub":"91'110","last_scrub_stamp":"2024-09-13T11:33:49.947781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:49.947781+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":111,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:13.777370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655572,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":7994,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"99'55","reported_seq":338,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.288164+0000","last_change":"2024-09-13T11:33:51.685010+0000","last_active":"2024-09-13T11:34:01.288164+0000","last_peered":"2024-09-13T11:34:01.288164+0000","last_clean":"2024-09-13T11:34:01.288164+0000","last_became_active":"2024-09-13T11:33:51.295188+0000","last_became_peered":"2024-09-13T11:33:51.295188+0000","last_unstale":"2024-09-13T11:34:01.288164+0000","last_undegraded":"2024-09-13T11:34:01.288164+0000","last_fullsized":"2024-09-13T11:34:01.288164+0000","mapping_epoch":92,"log_start":"98'52","ondisk_log_start":"98'52","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":5,"last_scrub":"93'46","last_scrub_stamp":"2024-09-13T11:33:51.684945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:51.684945+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":52,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:59.353266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6706125,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":97967,"num_write":10,"num_write_kb":3186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,10],"acting":[4,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"95'94","reported_seq":382,"reported_epoch":101,"state":"active+remapped","last_fresh":"2024-09-13T11:34:01.285720+0000","last_change":"2024-09-13T11:33:49.508610+0000","last_active":"2024-09-13T11:34:01.285720+0000","last_peered":"2024-09-13T11:34:01.285720+0000","last_clean":"2024-09-13T11:33:46.820878+0000","last_became_active":"2024-09-13T11:33:49.497339+0000","last_became_peered":"2024-09-13T11:33:49.497339+0000","last_unstale":"2024-09-13T11:34:01.285720+0000","last_undegraded":"2024-09-13T11:34:01.285720+0000","last_fullsized":"2024-09-13T11:34:01.285720+0000","mapping_epoch":90,"log_start":"95'91","ondisk_log_start":"95'91","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":91,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1487469,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":128,"num_read_kb":105955,"num_write":61,"num_write_kb":17387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3687913,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,10],"acting":[0,11,4],"avail_no_missing":["0","4","11"],"object_location_counts":[{"shards":"0,4,11","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"86'77","reported_seq":338,"reported_epoch":96,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:56.676867+0000","last_change":"2024-09-13T11:33:51.861069+0000","last_active":"2024-09-13T11:33:56.676867+0000","last_peered":"2024-09-13T11:33:56.676867+0000","last_clean":"2024-09-13T11:33:56.676867+0000","last_became_active":"2024-09-13T11:33:51.295757+0000","last_became_peered":"2024-09-13T11:33:51.295757+0000","last_unstale":"2024-09-13T11:33:56.676867+0000","last_undegraded":"2024-09-13T11:33:56.676867+0000","last_fullsized":"2024-09-13T11:33:56.676867+0000","mapping_epoch":92,"log_start":"84'74","ondisk_log_start":"84'74","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":5,"last_scrub":"86'77","last_scrub_stamp":"2024-09-13T11:33:51.860972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:51.860972+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":74,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:05.401717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,14],"acting":[6,14,7],"avail_no_missing":["6","7","14"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"53'20","reported_seq":236,"reported_epoch":101,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:34:01.284349+0000","last_change":"2024-09-13T11:33:12.028705+0000","last_active":"2024-09-13T11:34:01.284349+0000","last_peered":"2024-09-13T11:34:01.284349+0000","last_clean":"2024-09-13T11:34:01.284349+0000","last_became_active":"2024-09-13T11:32:59.281878+0000","last_became_peered":"2024-09-13T11:32:59.281878+0000","last_unstale":"2024-09-13T11:34:01.284349+0000","last_undegraded":"2024-09-13T11:34:01.284349+0000","last_fullsized":"2024-09-13T11:34:01.284349+0000","mapping_epoch":61,"log_start":"51'17","ondisk_log_start":"51'17","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'20","last_scrub_stamp":"2024-09-13T11:33:12.028588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.028588+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:21.615654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6,0,7],"avail_no_missing":["6","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"101'107","reported_seq":414,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:02.241069+0000","last_change":"2024-09-13T11:33:16.278769+0000","last_active":"2024-09-13T11:34:02.241069+0000","last_peered":"2024-09-13T11:34:02.241069+0000","last_clean":"2024-09-13T11:34:02.241069+0000","last_became_active":"2024-09-13T11:33:16.278471+0000","last_became_peered":"2024-09-13T11:33:16.278471+0000","last_unstale":"2024-09-13T11:34:02.241069+0000","last_undegraded":"2024-09-13T11:34:02.241069+0000","last_fullsized":"2024-09-13T11:34:02.241069+0000","mapping_epoch":72,"log_start":"101'104","ondisk_log_start":"101'104","created":56,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'57","last_scrub_stamp":"2024-09-13T11:33:04.555283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.555283+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":102,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:28.931440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6991844,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":137,"num_read_kb":115737,"num_write":59,"num_write_kb":16101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,6],"acting":[11,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":110,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.286191+0000","last_change":"2024-09-13T11:33:03.196131+0000","last_active":"2024-09-13T11:34:01.286191+0000","last_peered":"2024-09-13T11:34:01.286191+0000","last_clean":"2024-09-13T11:34:01.286191+0000","last_became_active":"2024-09-13T11:32:56.278133+0000","last_became_peered":"2024-09-13T11:32:56.278133+0000","last_unstale":"2024-09-13T11:34:01.286191+0000","last_undegraded":"2024-09-13T11:34:01.286191+0000","last_fullsized":"2024-09-13T11:34:01.286191+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.196058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:30.650376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6],"acting":[14,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.16","version":"100'104","reported_seq":415,"reported_epoch":101,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:34:02.200851+0000","last_change":"2024-09-13T11:32:56.273159+0000","last_active":"2024-09-13T11:34:02.200851+0000","last_peered":"2024-09-13T11:34:02.200851+0000","last_clean":"2024-09-13T11:32:55.237210+0000","last_became_active":"2024-09-13T11:32:56.273159+0000","last_became_peered":"2024-09-13T11:32:56.273159+0000","last_unstale":"2024-09-13T11:34:02.200851+0000","last_undegraded":"2024-09-13T11:32:56.268297+0000","last_fullsized":"2024-09-13T11:32:56.268083+0000","mapping_epoch":58,"log_start":"100'101","ondisk_log_start":"100'101","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":100,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:34:04.961344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5571636,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":240,"num_read_kb":209948,"num_write":84,"num_write_kb":25499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"94'52","reported_seq":310,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.285796+0000","last_change":"2024-09-13T11:33:05.991060+0000","last_active":"2024-09-13T11:34:01.285796+0000","last_peered":"2024-09-13T11:34:01.285796+0000","last_clean":"2024-09-13T11:34:01.285796+0000","last_became_active":"2024-09-13T11:32:59.285266+0000","last_became_peered":"2024-09-13T11:32:59.285266+0000","last_unstale":"2024-09-13T11:34:01.285796+0000","last_undegraded":"2024-09-13T11:34:01.285796+0000","last_fullsized":"2024-09-13T11:34:01.285796+0000","mapping_epoch":61,"log_start":"82'49","ondisk_log_start":"82'49","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'41","last_scrub_stamp":"2024-09-13T11:33:05.990981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.990981+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":46,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:26.585649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016906,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":24306,"num_write":10,"num_write_kb":2641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,6],"acting":[0,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"101'128","reported_seq":480,"reported_epoch":101,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:34:02.131425+0000","last_change":"2024-09-13T11:33:04.630617+0000","last_active":"2024-09-13T11:34:02.131425+0000","last_peered":"2024-09-13T11:34:02.131425+0000","last_clean":"2024-09-13T11:34:02.131425+0000","last_became_active":"2024-09-13T11:32:58.287813+0000","last_became_peered":"2024-09-13T11:32:58.287813+0000","last_unstale":"2024-09-13T11:34:02.131425+0000","last_undegraded":"2024-09-13T11:34:02.131425+0000","last_fullsized":"2024-09-13T11:34:02.131425+0000","mapping_epoch":60,"log_start":"101'125","ondisk_log_start":"101'125","created":56,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'78","last_scrub_stamp":"2024-09-13T11:33:04.630512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:04.630512+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:18.444394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5865775,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":172,"num_read_kb":144504,"num_write":55,"num_write_kb":14627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,3],"avail_no_missing":["4","3","7"],"object_location_counts":[{"shards":"3,4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"53'41","reported_seq":268,"reported_epoch":98,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:58.561291+0000","last_change":"2024-09-13T11:33:58.561198+0000","last_active":"2024-09-13T11:33:58.561291+0000","last_peered":"2024-09-13T11:33:58.561291+0000","last_clean":"2024-09-13T11:33:58.561291+0000","last_became_active":"2024-09-13T11:33:58.558145+0000","last_became_peered":"2024-09-13T11:33:58.558145+0000","last_unstale":"2024-09-13T11:33:58.561291+0000","last_undegraded":"2024-09-13T11:33:58.561291+0000","last_fullsized":"2024-09-13T11:33:58.561291+0000","mapping_epoch":97,"log_start":"50'38","ondisk_log_start":"50'38","created":56,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'41","last_scrub_stamp":"2024-09-13T11:33:05.531507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.531507+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:20.087860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13],"acting":[7,14,6],"avail_no_missing":["7","6","13","14"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"99'149","reported_seq":524,"reported_epoch":101,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:34:02.203578+0000","last_change":"2024-09-13T11:33:05.196910+0000","last_active":"2024-09-13T11:34:02.203578+0000","last_peered":"2024-09-13T11:34:02.203578+0000","last_clean":"2024-09-13T11:34:02.203578+0000","last_became_active":"2024-09-13T11:32:59.288348+0000","last_became_peered":"2024-09-13T11:32:59.288348+0000","last_unstale":"2024-09-13T11:34:02.203578+0000","last_undegraded":"2024-09-13T11:34:02.203578+0000","last_fullsized":"2024-09-13T11:34:02.203578+0000","mapping_epoch":61,"log_start":"98'146","ondisk_log_start":"98'146","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'93","last_scrub_stamp":"2024-09-13T11:33:05.196778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:05.196778+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":146,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:25.689022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6874162,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":192,"num_read_kb":177577,"num_write":66,"num_write_kb":18673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8],"acting":[14,8,3],"avail_no_missing":["14","3","8"],"object_location_counts":[{"shards":"3,8,14","objects":2}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"96'161","reported_seq":561,"reported_epoch":98,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:33:58.550152+0000","last_change":"2024-09-13T11:33:03.598922+0000","last_active":"2024-09-13T11:33:58.550152+0000","last_peered":"2024-09-13T11:33:58.550152+0000","last_clean":"2024-09-13T11:33:58.550152+0000","last_became_active":"2024-09-13T11:32:57.287130+0000","last_became_peered":"2024-09-13T11:32:57.287130+0000","last_unstale":"2024-09-13T11:33:58.550152+0000","last_undegraded":"2024-09-13T11:33:58.550152+0000","last_fullsized":"2024-09-13T11:33:58.550152+0000","mapping_epoch":59,"log_start":"94'158","ondisk_log_start":"94'158","created":56,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'103","last_scrub_stamp":"2024-09-13T11:33:03.598827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.598827+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":157,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:28.310675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11244147,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":203,"num_read_kb":221283,"num_write":59,"num_write_kb":20853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,15],"avail_no_missing":["7","3","15"],"object_location_counts":[{"shards":"3,7,15","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"101'76","reported_seq":371,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:02.261222+0000","last_change":"2024-09-13T11:33:12.208901+0000","last_active":"2024-09-13T11:34:02.261222+0000","last_peered":"2024-09-13T11:34:02.261222+0000","last_clean":"2024-09-13T11:34:02.261222+0000","last_became_active":"2024-09-13T11:32:59.280441+0000","last_became_peered":"2024-09-13T11:32:59.280441+0000","last_unstale":"2024-09-13T11:34:02.261222+0000","last_undegraded":"2024-09-13T11:34:02.261222+0000","last_fullsized":"2024-09-13T11:34:02.261222+0000","mapping_epoch":61,"log_start":"101'73","ondisk_log_start":"101'73","created":56,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"66'60","last_scrub_stamp":"2024-09-13T11:33:12.208834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:12.208834+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":70,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:31.790218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3697606,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":65,"num_read_kb":56463,"num_write":20,"num_write_kb":5819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,3],"acting":[10,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":189,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:33:58.559426+0000","last_change":"2024-09-13T11:33:58.559324+0000","last_active":"2024-09-13T11:33:58.559426+0000","last_peered":"2024-09-13T11:33:58.559426+0000","last_clean":"2024-09-13T11:33:58.559426+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:33:58.559426+0000","last_undegraded":"2024-09-13T11:33:58.559426+0000","last_fullsized":"2024-09-13T11:33:58.559426+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:53.031720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:53.031720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:54.145954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.1","version":"85'126","reported_seq":489,"reported_epoch":97,"state":"remapped+peering","last_fresh":"2024-09-13T11:33:57.553252+0000","last_change":"2024-09-13T11:33:58.548150+0000","last_active":"2024-09-13T11:33:57.553252+0000","last_peered":"2024-09-13T11:33:56.893153+0000","last_clean":"2024-09-13T11:33:56.893153+0000","last_became_active":"2024-09-13T11:33:46.493765+0000","last_became_peered":"2024-09-13T11:33:46.493765+0000","last_unstale":"2024-09-13T11:33:57.553252+0000","last_undegraded":"2024-09-13T11:33:57.553252+0000","last_fullsized":"2024-09-13T11:33:57.553252+0000","mapping_epoch":98,"log_start":"84'123","ondisk_log_start":"84'123","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'126","last_scrub_stamp":"2024-09-13T11:33:48.012476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:48.012476+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":121,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3,13],"acting":[15,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7,13],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":175,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.286207+0000","last_change":"2024-09-13T11:33:39.971540+0000","last_active":"2024-09-13T11:34:01.286207+0000","last_peered":"2024-09-13T11:34:01.286207+0000","last_clean":"2024-09-13T11:34:01.286207+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:34:01.286207+0000","last_undegraded":"2024-09-13T11:34:01.286207+0000","last_fullsized":"2024-09-13T11:34:01.286207+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:39.971482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:39.971482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:55.187895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"95'92","reported_seq":419,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.288349+0000","last_change":"2024-09-13T11:33:47.984997+0000","last_active":"2024-09-13T11:34:01.288349+0000","last_peered":"2024-09-13T11:34:01.288349+0000","last_clean":"2024-09-13T11:34:01.288349+0000","last_became_active":"2024-09-13T11:33:46.495057+0000","last_became_peered":"2024-09-13T11:33:46.495057+0000","last_unstale":"2024-09-13T11:34:01.288349+0000","last_undegraded":"2024-09-13T11:34:01.288349+0000","last_fullsized":"2024-09-13T11:34:01.288349+0000","mapping_epoch":51,"log_start":"95'89","ondisk_log_start":"95'89","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'88","last_scrub_stamp":"2024-09-13T11:33:47.984919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:47.984919+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":89,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:54.642259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3277431,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":51824,"num_write":5,"num_write_kb":1313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,8],"acting":[10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":171,"reported_epoch":101,"state":"active+undersized","last_fresh":"2024-09-13T11:34:01.668258+0000","last_change":"2024-09-13T11:32:32.498855+0000","last_active":"2024-09-13T11:34:01.668258+0000","last_peered":"2024-09-13T11:34:01.668258+0000","last_clean":"2024-09-13T11:32:30.479480+0000","last_became_active":"2024-09-13T11:32:32.498855+0000","last_became_peered":"2024-09-13T11:32:32.498855+0000","last_unstale":"2024-09-13T11:34:01.668258+0000","last_undegraded":"2024-09-13T11:34:01.668258+0000","last_fullsized":"2024-09-13T11:32:32.497672+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":["8","6"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"86'69","reported_seq":299,"reported_epoch":101,"state":"active+undersized","last_fresh":"2024-09-13T11:34:01.289396+0000","last_change":"2024-09-13T11:33:46.486590+0000","last_active":"2024-09-13T11:34:01.289396+0000","last_peered":"2024-09-13T11:34:01.289396+0000","last_clean":"2024-09-13T11:32:49.952102+0000","last_became_active":"2024-09-13T11:33:46.486590+0000","last_became_peered":"2024-09-13T11:33:46.486590+0000","last_unstale":"2024-09-13T11:34:01.289396+0000","last_undegraded":"2024-09-13T11:34:01.289396+0000","last_fullsized":"2024-09-13T11:33:46.482126+0000","mapping_epoch":59,"log_start":"85'66","ondisk_log_start":"85'66","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":66,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:34:04.930247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1218557,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":24778,"num_write":26,"num_write_kb":8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":1191591,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6],"acting":[11,6],"avail_no_missing":["11","6"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":170,"reported_epoch":101,"state":"active+undersized","last_fresh":"2024-09-13T11:34:01.284497+0000","last_change":"2024-09-13T11:32:57.283324+0000","last_active":"2024-09-13T11:34:01.284497+0000","last_peered":"2024-09-13T11:34:01.284497+0000","last_clean":"2024-09-13T11:32:55.276320+0000","last_became_active":"2024-09-13T11:32:57.283324+0000","last_became_peered":"2024-09-13T11:32:57.283324+0000","last_unstale":"2024-09-13T11:34:01.284497+0000","last_undegraded":"2024-09-13T11:34:01.284497+0000","last_fullsized":"2024-09-13T11:32:57.273798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:32:05.330509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"101'153","reported_seq":492,"reported_epoch":101,"state":"active+undersized+remapped","last_fresh":"2024-09-13T11:34:02.301645+0000","last_change":"2024-09-13T11:33:46.486407+0000","last_active":"2024-09-13T11:34:02.301645+0000","last_peered":"2024-09-13T11:34:02.301645+0000","last_clean":"2024-09-13T11:32:55.273515+0000","last_became_active":"2024-09-13T11:33:46.486407+0000","last_became_peered":"2024-09-13T11:33:46.486407+0000","last_unstale":"2024-09-13T11:34:02.301645+0000","last_undegraded":"2024-09-13T11:34:02.301645+0000","last_fullsized":"2024-09-13T11:33:46.483345+0000","mapping_epoch":59,"log_start":"101'150","ondisk_log_start":"101'150","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":147,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:34:03.046321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243858,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":139,"num_read_kb":118050,"num_write":65,"num_write_kb":19601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2250639,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3,9],"avail_no_missing":["3","9"],"object_location_counts":[{"shards":"3,9","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"96'343","reported_seq":572,"reported_epoch":96,"state":"remapped+peering","last_fresh":"2024-09-13T11:33:56.957407+0000","last_change":"2024-09-13T11:33:58.548213+0000","last_active":"2024-09-13T11:33:58.548213+0000","last_peered":"2024-09-13T11:33:56.957407+0000","last_clean":"2024-09-13T11:33:56.957407+0000","last_became_active":"2024-09-13T11:32:48.281772+0000","last_became_peered":"2024-09-13T11:32:48.281772+0000","last_unstale":"2024-09-13T11:33:56.957407+0000","last_undegraded":"2024-09-13T11:33:56.957407+0000","last_fullsized":"2024-09-13T11:33:56.957407+0000","mapping_epoch":98,"log_start":"96'339","ondisk_log_start":"96'339","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'252","last_scrub_stamp":"2024-09-13T11:33:29.491435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:33:29.491435+0000","objects_scrubbed":42,"log_size":4,"log_dups_size":348,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23748052,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":307,"num_read_kb":9796,"num_write":449,"num_write_kb":86350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":14112815,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7],"acting":[13,7,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7,15],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":187,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.286805+0000","last_change":"2024-09-13T11:33:26.770230+0000","last_active":"2024-09-13T11:34:01.286805+0000","last_peered":"2024-09-13T11:34:01.286805+0000","last_clean":"2024-09-13T11:34:01.286805+0000","last_became_active":"2024-09-13T11:32:43.554077+0000","last_became_peered":"2024-09-13T11:32:43.554077+0000","last_unstale":"2024-09-13T11:34:01.286805+0000","last_undegraded":"2024-09-13T11:34:01.286805+0000","last_fullsized":"2024-09-13T11:34:01.286805+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:33:26.770137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:26.770137+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:51.278083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,11],"acting":[3,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"99'90","reported_seq":393,"reported_epoch":101,"state":"active+clean+remapped","last_fresh":"2024-09-13T11:34:01.287004+0000","last_change":"2024-09-13T11:33:01.558907+0000","last_active":"2024-09-13T11:34:01.287004+0000","last_peered":"2024-09-13T11:34:01.287004+0000","last_clean":"2024-09-13T11:34:01.287004+0000","last_became_active":"2024-09-13T11:32:54.600364+0000","last_became_peered":"2024-09-13T11:32:54.600364+0000","last_unstale":"2024-09-13T11:34:01.287004+0000","last_undegraded":"2024-09-13T11:34:01.287004+0000","last_fullsized":"2024-09-13T11:34:01.287004+0000","mapping_epoch":49,"log_start":"99'87","ondisk_log_start":"99'87","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'73","last_scrub_stamp":"2024-09-13T11:33:01.558809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.558809+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":84,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:08.259528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512729,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":90416,"num_write":19,"num_write_kb":5237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8],"acting":[3,8,5],"avail_no_missing":["3","5","8"],"object_location_counts":[{"shards":"3,5,8","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"98'339","reported_seq":574,"reported_epoch":98,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:58.574112+0000","last_change":"2024-09-13T11:32:57.280350+0000","last_active":"2024-09-13T11:33:58.574112+0000","last_peered":"2024-09-13T11:33:58.574112+0000","last_clean":"2024-09-13T11:32:54.594742+0000","last_became_active":"2024-09-13T11:32:57.280350+0000","last_became_peered":"2024-09-13T11:32:57.280350+0000","last_unstale":"2024-09-13T11:33:58.574112+0000","last_undegraded":"2024-09-13T11:32:57.275975+0000","last_fullsized":"2024-09-13T11:32:57.275775+0000","mapping_epoch":59,"log_start":"97'335","ondisk_log_start":"97'335","created":30,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":341,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:33:59.998401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35021365,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":307,"num_read_kb":13586,"num_write":442,"num_write_kb":93287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":24388633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6],"acting":[15,6],"avail_no_missing":["15","6"],"object_location_counts":[{"shards":"6,15","objects":45}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:33:58.566481+0000","last_change":"2024-09-13T11:33:58.566068+0000","last_active":"2024-09-13T11:33:58.566481+0000","last_peered":"2024-09-13T11:33:58.566481+0000","last_clean":"2024-09-13T11:33:58.566481+0000","last_became_active":"2024-09-13T11:33:58.565800+0000","last_became_peered":"2024-09-13T11:33:58.565800+0000","last_unstale":"2024-09-13T11:33:58.566481+0000","last_undegraded":"2024-09-13T11:33:58.566481+0000","last_fullsized":"2024-09-13T11:33:58.566481+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:28.020512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:28.020512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:41.792598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,14,4],"acting":[13,14,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":193,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.291255+0000","last_change":"2024-09-13T11:33:58.557296+0000","last_active":"2024-09-13T11:34:01.291255+0000","last_peered":"2024-09-13T11:34:01.291255+0000","last_clean":"2024-09-13T11:34:01.291255+0000","last_became_active":"2024-09-13T11:33:58.556908+0000","last_became_peered":"2024-09-13T11:33:58.556908+0000","last_unstale":"2024-09-13T11:34:01.291255+0000","last_undegraded":"2024-09-13T11:34:01.291255+0000","last_fullsized":"2024-09-13T11:34:01.291255+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:03.236001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:03.236001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:09.527162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,13,6],"acting":[10,13,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"101'387","reported_seq":655,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:02.171798+0000","last_change":"2024-09-13T11:33:40.953610+0000","last_active":"2024-09-13T11:34:02.171798+0000","last_peered":"2024-09-13T11:34:02.171798+0000","last_clean":"2024-09-13T11:34:02.171798+0000","last_became_active":"2024-09-13T11:32:47.277392+0000","last_became_peered":"2024-09-13T11:32:47.277392+0000","last_unstale":"2024-09-13T11:34:02.171798+0000","last_undegraded":"2024-09-13T11:34:02.171798+0000","last_fullsized":"2024-09-13T11:34:02.171798+0000","mapping_epoch":51,"log_start":"101'384","ondisk_log_start":"101'384","created":30,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'333","last_scrub_stamp":"2024-09-13T11:33:40.953425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:33:40.953425+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":399,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:54.797020+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":342,"num_read_kb":6439,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":28975184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,0],"acting":[6,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":174,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.286389+0000","last_change":"2024-09-13T11:33:35.914912+0000","last_active":"2024-09-13T11:34:01.286389+0000","last_peered":"2024-09-13T11:34:01.286389+0000","last_clean":"2024-09-13T11:34:01.286389+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:34:01.286389+0000","last_undegraded":"2024-09-13T11:34:01.286389+0000","last_fullsized":"2024-09-13T11:34:01.286389+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:33:35.914850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:35.914850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:50.604726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"101'105","reported_seq":428,"reported_epoch":101,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:34:02.299908+0000","last_change":"2024-09-13T11:32:57.280636+0000","last_active":"2024-09-13T11:34:02.299908+0000","last_peered":"2024-09-13T11:34:02.299908+0000","last_clean":"2024-09-13T11:32:54.904504+0000","last_became_active":"2024-09-13T11:32:57.280636+0000","last_became_peered":"2024-09-13T11:32:57.280636+0000","last_unstale":"2024-09-13T11:34:02.299908+0000","last_undegraded":"2024-09-13T11:32:57.277736+0000","last_fullsized":"2024-09-13T11:32:57.277594+0000","mapping_epoch":59,"log_start":"101'102","ondisk_log_start":"101'102","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":100,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:34:06.972311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7033878,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":265,"num_read_kb":202676,"num_write":78,"num_write_kb":23995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6],"acting":[10,6],"avail_no_missing":["10","6"],"object_location_counts":[{"shards":"6,10","objects":2}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"94'392","reported_seq":610,"reported_epoch":94,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:53.883221+0000","last_change":"2024-09-13T11:33:16.277928+0000","last_active":"2024-09-13T11:33:53.883221+0000","last_peered":"2024-09-13T11:33:53.883221+0000","last_clean":"2024-09-13T11:32:55.257840+0000","last_became_active":"2024-09-13T11:33:16.277928+0000","last_became_peered":"2024-09-13T11:33:16.277928+0000","last_unstale":"2024-09-13T11:33:53.883221+0000","last_undegraded":"2024-09-13T11:33:16.272371+0000","last_fullsized":"2024-09-13T11:33:16.272253+0000","mapping_epoch":72,"log_start":"93'389","ondisk_log_start":"93'389","created":30,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:32:18.407384+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":404,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19852239,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":352,"num_read_kb":5465,"num_write":495,"num_write_kb":96860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":16251716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,3],"acting":[15,3],"avail_no_missing":["15","3"],"object_location_counts":[{"shards":"3,15","objects":24}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":155,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-13T11:33:58.557435+0000","last_change":"2024-09-13T11:33:58.557336+0000","last_active":"2024-09-13T11:33:58.557435+0000","last_peered":"2024-09-13T11:33:58.557435+0000","last_clean":"2024-09-13T11:33:58.557435+0000","last_became_active":"2024-09-13T11:33:58.554112+0000","last_became_peered":"2024-09-13T11:33:58.554112+0000","last_unstale":"2024-09-13T11:33:58.557435+0000","last_undegraded":"2024-09-13T11:33:58.557435+0000","last_fullsized":"2024-09-13T11:33:58.557435+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:33:27.080847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:33:27.080847+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:48.074760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,8],"acting":[15,13,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":275,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-13T11:33:56.922429+0000","last_change":"2024-09-13T11:33:16.944043+0000","last_active":"2024-09-13T11:33:56.922429+0000","last_peered":"2024-09-13T11:33:56.922429+0000","last_clean":"2024-09-13T11:33:56.922429+0000","last_became_active":"2024-09-13T11:32:48.286550+0000","last_became_peered":"2024-09-13T11:32:48.286550+0000","last_unstale":"2024-09-13T11:33:56.922429+0000","last_undegraded":"2024-09-13T11:33:56.922429+0000","last_fullsized":"2024-09-13T11:33:56.922429+0000","mapping_epoch":52,"log_start":"25'109","ondisk_log_start":"25'109","created":23,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:33:16.943930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:33:16.943930+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:23.725756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,8],"acting":[0,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"98'52","reported_seq":306,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.286876+0000","last_change":"2024-09-13T11:33:02.151042+0000","last_active":"2024-09-13T11:34:01.286876+0000","last_peered":"2024-09-13T11:34:01.286876+0000","last_clean":"2024-09-13T11:34:01.286876+0000","last_became_active":"2024-09-13T11:32:54.609316+0000","last_became_peered":"2024-09-13T11:32:54.609316+0000","last_unstale":"2024-09-13T11:34:01.286876+0000","last_undegraded":"2024-09-13T11:34:01.286876+0000","last_fullsized":"2024-09-13T11:34:01.286876+0000","mapping_epoch":50,"log_start":"97'49","ondisk_log_start":"97'49","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'41","last_scrub_stamp":"2024-09-13T11:33:02.150975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.150975+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":48,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:06.638916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":42562,"num_write":10,"num_write_kb":2812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.8","version":"94'58","reported_seq":325,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.283736+0000","last_change":"2024-09-13T11:33:01.646865+0000","last_active":"2024-09-13T11:34:01.283736+0000","last_peered":"2024-09-13T11:34:01.283736+0000","last_clean":"2024-09-13T11:34:01.283736+0000","last_became_active":"2024-09-13T11:32:54.601608+0000","last_became_peered":"2024-09-13T11:32:54.601608+0000","last_unstale":"2024-09-13T11:34:01.283736+0000","last_undegraded":"2024-09-13T11:34:01.283736+0000","last_fullsized":"2024-09-13T11:34:01.283736+0000","mapping_epoch":31,"log_start":"94'55","ondisk_log_start":"94'55","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'53","last_scrub_stamp":"2024-09-13T11:33:01.646779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:01.646779+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":54,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:19.033394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2974905,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":103617,"num_write":5,"num_write_kb":1252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"101'71","reported_seq":346,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:02.194645+0000","last_change":"2024-09-13T11:33:02.042709+0000","last_active":"2024-09-13T11:34:02.194645+0000","last_peered":"2024-09-13T11:34:02.194645+0000","last_clean":"2024-09-13T11:34:02.194645+0000","last_became_active":"2024-09-13T11:32:54.604804+0000","last_became_peered":"2024-09-13T11:32:54.604804+0000","last_unstale":"2024-09-13T11:34:02.194645+0000","last_undegraded":"2024-09-13T11:34:02.194645+0000","last_fullsized":"2024-09-13T11:34:02.194645+0000","mapping_epoch":31,"log_start":"101'68","ondisk_log_start":"101'68","created":31,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'28","last_scrub_stamp":"2024-09-13T11:33:02.042611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:02.042611+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:17.971364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7059072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":118359,"num_write":42,"num_write_kb":12843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,14,7],"acting":[0,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"96'88","reported_seq":371,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:02.241491+0000","last_change":"2024-09-13T11:33:47.872537+0000","last_active":"2024-09-13T11:34:02.241491+0000","last_peered":"2024-09-13T11:34:02.241491+0000","last_clean":"2024-09-13T11:34:02.241491+0000","last_became_active":"2024-09-13T11:33:46.493697+0000","last_became_peered":"2024-09-13T11:33:46.493697+0000","last_unstale":"2024-09-13T11:34:02.241491+0000","last_undegraded":"2024-09-13T11:34:02.241491+0000","last_fullsized":"2024-09-13T11:34:02.241491+0000","mapping_epoch":51,"log_start":"96'85","ondisk_log_start":"96'85","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'77","last_scrub_stamp":"2024-09-13T11:33:47.872461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:47.872461+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":85,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:52.244421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6302724,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":18606,"num_write":14,"num_write_kb":4476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"86'79","reported_seq":348,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-13T11:33:56.882258+0000","last_change":"2024-09-13T11:33:49.900984+0000","last_active":"2024-09-13T11:33:56.882258+0000","last_peered":"2024-09-13T11:33:56.882258+0000","last_clean":"2024-09-13T11:33:56.882258+0000","last_became_active":"2024-09-13T11:33:46.494713+0000","last_became_peered":"2024-09-13T11:33:46.494713+0000","last_unstale":"2024-09-13T11:33:56.882258+0000","last_undegraded":"2024-09-13T11:33:56.882258+0000","last_fullsized":"2024-09-13T11:33:56.882258+0000","mapping_epoch":52,"log_start":"83'76","ondisk_log_start":"83'76","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'79","last_scrub_stamp":"2024-09-13T11:33:49.900926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:49.900926+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":76,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:52.479638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2000099,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'44","reported_seq":273,"reported_epoch":98,"state":"active+undersized+degraded","last_fresh":"2024-09-13T11:33:58.547772+0000","last_change":"2024-09-13T11:33:46.493682+0000","last_active":"2024-09-13T11:33:58.547772+0000","last_peered":"2024-09-13T11:33:58.547772+0000","last_clean":"2024-09-13T11:32:55.276844+0000","last_became_active":"2024-09-13T11:33:46.493682+0000","last_became_peered":"2024-09-13T11:33:46.493682+0000","last_unstale":"2024-09-13T11:33:58.547772+0000","last_undegraded":"2024-09-13T11:33:46.488168+0000","last_fullsized":"2024-09-13T11:33:46.488048+0000","mapping_epoch":59,"log_start":"84'41","ondisk_log_start":"84'41","created":31,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":41,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-13T11:34:01.009387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336239,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":40308,"num_write":21,"num_write_kb":6754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":1020412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7],"acting":[15,7],"avail_no_missing":["15","7"],"object_location_counts":[{"shards":"7,15","objects":1}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.d","version":"101'109","reported_seq":427,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:01.283807+0000","last_change":"2024-09-13T11:33:47.707289+0000","last_active":"2024-09-13T11:34:01.283807+0000","last_peered":"2024-09-13T11:34:01.283807+0000","last_clean":"2024-09-13T11:34:01.283807+0000","last_became_active":"2024-09-13T11:33:46.496028+0000","last_became_peered":"2024-09-13T11:33:46.496028+0000","last_unstale":"2024-09-13T11:34:01.283807+0000","last_undegraded":"2024-09-13T11:34:01.283807+0000","last_fullsized":"2024-09-13T11:34:01.283807+0000","mapping_epoch":72,"log_start":"85'102","ondisk_log_start":"85'102","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'105","last_scrub_stamp":"2024-09-13T11:33:47.707205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:47.707205+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":102,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:34:58.813325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6638864,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":18589,"num_write":4,"num_write_kb":1272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"95'119","reported_seq":482,"reported_epoch":97,"state":"remapped+peering","last_fresh":"2024-09-13T11:33:57.552836+0000","last_change":"2024-09-13T11:33:58.548462+0000","last_active":"2024-09-13T11:33:57.552836+0000","last_peered":"2024-09-13T11:33:57.513379+0000","last_clean":"2024-09-13T11:33:57.513379+0000","last_became_active":"2024-09-13T11:33:46.494884+0000","last_became_peered":"2024-09-13T11:33:46.494884+0000","last_unstale":"2024-09-13T11:33:57.552836+0000","last_undegraded":"2024-09-13T11:33:57.552836+0000","last_fullsized":"2024-09-13T11:33:57.552836+0000","mapping_epoch":98,"log_start":"94'116","ondisk_log_start":"94'116","created":31,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'109","last_scrub_stamp":"2024-09-13T11:33:49.040004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:33:49.040004+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":113,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832161,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":14845,"num_write":10,"num_write_kb":2504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,13],"acting":[15,14,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,13,14],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.f","version":"101'154","reported_seq":565,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-13T11:34:02.268077+0000","last_change":"2024-09-13T11:34:01.295347+0000","last_active":"2024-09-13T11:34:02.268077+0000","last_peered":"2024-09-13T11:34:02.268077+0000","last_clean":"2024-09-13T11:34:02.268077+0000","last_became_active":"2024-09-13T11:34:01.294989+0000","last_became_peered":"2024-09-13T11:34:01.294989+0000","last_unstale":"2024-09-13T11:34:02.268077+0000","last_undegraded":"2024-09-13T11:34:02.268077+0000","last_fullsized":"2024-09-13T11:34:02.268077+0000","mapping_epoch":100,"log_start":"101'151","ondisk_log_start":"101'151","created":31,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:19.415426+0000","last_clean_scrub_stamp":"2024-09-13T11:32:19.415426+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":151,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075895,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":262,"num_read_kb":259986,"num_write":90,"num_write_kb":24991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8390090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,13],"acting":[10,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":134979912,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":2586,"num_read_kb":2377604,"num_write":836,"num_write_kb":247577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":21462266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":17,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":310112256,"data_stored":309639236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":91,"acting":103,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":107390064,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":69,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":1308,"num_read_kb":35286,"num_write":1881,"num_write_kb":379843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":83728348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":390774784,"data_stored":389716857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":9,"acting":10,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":22,"acting":22,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247835,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":123492,"kb_used_data":92308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94248348,"statfs":{"total":96636764160,"available":96510308352,"internally_reserved":0,"allocated":94523392,"data_stored":93934682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59548,"kb_used_data":24204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94312292,"statfs":{"total":96636764160,"available":96575787008,"internally_reserved":0,"allocated":24784896,"data_stored":24360720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3,4,5,6,7,8,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247834,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81392,"kb_used_data":50208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94290448,"statfs":{"total":96636764160,"available":96553418752,"internally_reserved":0,"allocated":51412992,"data_stored":50957111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280538,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79164,"kb_used_data":43820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94292676,"statfs":{"total":96636764160,"available":96555700224,"internally_reserved":0,"allocated":44871680,"data_stored":44402918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,6,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280538,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280538,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54212,"kb_used_data":27188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94317628,"statfs":{"total":96636764160,"available":96581251072,"internally_reserved":0,"allocated":27840512,"data_stored":27448326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313223,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":170144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280539,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92256,"kb_used_data":61072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94279584,"statfs":{"total":96636764160,"available":96542294016,"internally_reserved":0,"allocated":62537728,"data_stored":62081620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,4,5,6,7,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280538,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54828,"kb_used_data":27804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94317012,"statfs":{"total":96636764160,"available":96580620288,"internally_reserved":0,"allocated":28471296,"data_stored":28089163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":94489280538,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280539,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113640,"kb_used_data":82456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94258200,"statfs":{"total":96636764160,"available":96520396800,"internally_reserved":0,"allocated":84434944,"data_stored":83865066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280539,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41212,"kb_used_data":14188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330628,"statfs":{"total":96636764160,"available":96594563072,"internally_reserved":0,"allocated":14528512,"data_stored":14156412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280538,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280539,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67544,"kb_used_data":36360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304296,"statfs":{"total":96636764160,"available":96567599104,"internally_reserved":0,"allocated":37232640,"data_stored":36816393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313242,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75984,"kb_used_data":44800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94295856,"statfs":{"total":96636764160,"available":96558956544,"internally_reserved":0,"allocated":45875200,"data_stored":45399780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,6,7,8,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95324,"kb_used_data":68300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276516,"statfs":{"total":96636764160,"available":96539152384,"internally_reserved":0,"allocated":69939200,"data_stored":69555056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24444928,"data_stored":24413026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":23801856,"data_stored":23781575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":64294912,"data_stored":64094881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":23842816,"data_stored":23748052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28405760,"data_stored":28348291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":42668032,"data_stored":42597076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":23842816,"data_stored":23748052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":79355904,"data_stored":79120996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17920000,"data_stored":17895427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18067456,"data_stored":18030231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4239360,"data_stored":4225584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40534016,"data_stored":40475943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29159424,"data_stored":29122319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2818048,"data_stored":2814409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":22638592,"data_stored":22604286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":27242496,"data_stored":27204695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13537280,"data_stored":13517632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4976640,"data_stored":4974330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":39108608,"data_stored":39042666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17780736,"data_stored":17753234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":26669056,"data_stored":26630598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":31195136,"data_stored":31146864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14225408,"data_stored":14201018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:34:04.679 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-13T11:34:04.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-13T11:34:04.680 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:04.680 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:34:04.680 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:04.688 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-13T11:34:04.737 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:34:04.736+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:34:04.791 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:34:04.791 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:34:04.792 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:04.800 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-13T11:34:04.838 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-13T11:34:04.837+0000 7f4eb1a4f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 32867) UID: 0 2024-09-13T11:34:04.939 INFO:tasks.ceph.osd.15.smithi204.stderr:2024-09-13T11:34:04.938+0000 7fb48993b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33343) UID: 0 2024-09-13T11:34:04.943 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:214 3.3 e38/30 3:daff7d1f:::smithi20438004-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.844575942999995, 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018230697000000001, 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.097172+0000", 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.11e-07 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109539+0000", 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01236655 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109558+0000", 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9225999999999999e-05 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109569+0000", 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1214e-05 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109680+0000", 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000110652 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109724+0000", 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4397000000000002e-05 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115334+0000", 2024-09-13T11:34:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056099540000000003 2024-09-13T11:34:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.115403+0000", 2024-09-13T11:34:04.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8492999999999996e-05 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:211 3.3 e38/30 3:de2fec76:::smithi20438004-12 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.832141797999995, 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022092034, 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109606+0000", 2024-09-13T11:34:04.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109612+0000", 2024-09-13T11:34:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4369999999999998e-06 2024-09-13T11:34:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119494+0000", 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0098819990000000007 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119523+0000", 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9295999999999999e-05 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119534+0000", 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0686999999999999e-05 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119598+0000", 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4085999999999996e-05 2024-09-13T11:34:04.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119617+0000", 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9687e-05 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131622+0000", 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012004989000000001 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131698+0000", 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5852999999999994e-05 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:212 3.3 e38/30 3:ff417594:::smithi20438004-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'3, mlcod=0'0)", 2024-09-13T11:34:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:34:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.822188204, 2024-09-13T11:34:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025717467000000001, 2024-09-13T11:34:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.957 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119560+0000", 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119561+0000", 2024-09-13T11:34:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.9600000000000008e-07 2024-09-13T11:34:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127225+0000", 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0076637759999999997 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127242+0000", 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.702e-05 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127259+0000", 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7007000000000002e-05 2024-09-13T11:34:04.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127335+0000", 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6341999999999995e-05 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.127379+0000", 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3816999999999997e-05 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145216+0000", 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017837664999999999 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.145277+0000", 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0844000000000002e-05 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:228 3.3 e38/30 3:c765c2c5:::smithi20438004-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=38'1)", 2024-09-13T11:34:04.963 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:34:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.791470396999998, 2024-09-13T11:34:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028575957999999999, 2024-09-13T11:34:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.150278+0000", 2024-09-13T11:34:04.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3e-07 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169049+0000", 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018770889999999998 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169067+0000", 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7810000000000001e-05 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169076+0000", 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1800000000000002e-06 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169126+0000", 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9505999999999997e-05 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169146+0000", 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0333e-05 2024-09-13T11:34:04.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178763+0000", 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0096172600000000007 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178854+0000", 2024-09-13T11:34:04.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0248999999999996e-05 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:226 3.3 e38/30 3:fd210b3a:::smithi20438004-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=38'2)", 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.772647078000006, 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.027022289000000001, 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.969 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169101+0000", 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169102+0000", 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6000000000000004e-07 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.970 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182563+0000", 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013461427999999999 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182584+0000", 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1036999999999999e-05 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182596+0000", 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.207e-05 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.971 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182658+0000", 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1705999999999995e-05 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182680+0000", 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196074+0000", 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013394540999999999 2024-09-13T11:34:04.972 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196123+0000", 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8801000000000003e-05 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.973 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:229 3.3 e38/30 3:d73fa370:::smithi20438004-30 low_tier:head v 38'7, mlcod=38'2)", 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.759127728999999, 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015438544, 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.974 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182620+0000", 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.182621+0000", 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8000000000000004e-07 2024-09-13T11:34:04.975 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192202+0000", 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095803799999999995 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192218+0000", 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6463000000000001e-05 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192230+0000", 2024-09-13T11:34:04.976 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2096000000000001e-05 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192303+0000", 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2793999999999995e-05 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192321+0000", 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.806e-05 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.977 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198007+0000", 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.005685659 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.198059+0000", 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2212000000000001e-05 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.978 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:238 3.3 e38/30 3:d3f658bf:::smithi20438004-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'3)", 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.749487396000006, 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019595936000000001, 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.979 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.980 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.192261+0000", 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3399999999999998e-07 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202505+0000", 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010243820000000001 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202522+0000", 2024-09-13T11:34:04.981 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6816999999999998e-05 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202531+0000", 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8200000000000003e-06 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.982 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202615+0000", 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3978000000000001e-05 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202637+0000", 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2243999999999999e-05 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211793+0000", 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0091556729999999996 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.983 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211857+0000", 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3849999999999993e-05 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:242 3.3 e38/30 3:f55e895c:::smithi20438004-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'3)", 2024-09-13T11:34:04.984 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.739181896000005, 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018936522000000001, 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.985 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202566+0000", 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.202567+0000", 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2610000000000001e-06 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.986 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212680+0000", 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010112335 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212692+0000", 2024-09-13T11:34:04.987 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1807000000000001e-05 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212698+0000", 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4760000000000003e-06 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212756+0000", 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7911000000000002e-05 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.988 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212779+0000", 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2648999999999999e-05 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221460+0000", 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.008681345 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.989 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221503+0000", 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2738000000000003e-05 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:250 3.3 e38/30 3:ff4c0fba:::smithi20438004-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'10, mlcod=38'7)", 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 97.728795954000006, 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013488022000000001, 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.990 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212952+0000", 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.991 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212953+0000", 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2399999999999996e-07 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219859+0000", 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0069059079999999997 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219870+0000", 2024-09-13T11:34:04.992 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0643e-05 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219877+0000", 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2359999999999998e-06 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219934+0000", 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7394e-05 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.993 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219955+0000", 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0684999999999999e-05 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226397+0000", 2024-09-13T11:34:04.994 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006441904 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.226440+0000", 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3328000000000003e-05 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:04.995 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:04.996 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.996 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.996 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:36 3.3 3:e2816a99:::d4bb749789c90c884ffd20ac62f804c7a6b80502d14432fa627e1239249ee2da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:04.996 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:34:04.996 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.671268554999997, 2024-09-13T11:34:04.996 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013305613000000001, 2024-09-13T11:34:04.996 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270479+0000", 2024-09-13T11:34:04.997 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270480+0000", 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2399999999999998e-07 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270655+0000", 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017481300000000001 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.998 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:04.999 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270683+0000", 2024-09-13T11:34:04.999 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8340999999999999e-05 2024-09-13T11:34:04.999 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:04.999 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:04.999 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270693+0000", 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6719999999999999e-06 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271029+0000", 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00033640200000000002 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.000 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271038+0000", 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6400000000000003e-06 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278355+0000", 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0073166510000000004 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278481+0000", 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000125955 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.001 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278657+0000", 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000176816 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280615+0000", 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0019576810000000002 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283538+0000", 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029229030000000001 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.002 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283592+0000", 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.4366000000000001e-05 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283671+0000", 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8337000000000006e-05 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283785+0000", 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000114412 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.003 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:40 3.3 3:c7a12847:::768a08ae37e9bf19369a517906ebd209c1f2b8ad3684596d644da7d48b42114d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.671025448999998, 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013347114, 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.004 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270723+0000", 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9100000000000004e-07 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.005 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270839+0000", 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000115369 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270847+0000", 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4309999999999994e-06 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270854+0000", 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.708e-06 2024-09-13T11:34:05.006 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271073+0000", 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00021964400000000001 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271080+0000", 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7909999999999999e-06 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.278928+0000", 2024-09-13T11:34:05.007 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0078479819999999999 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279058+0000", 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000130102 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.279300+0000", 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00024166300000000001 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.008 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280641+0000", 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0013409229999999999 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283545+0000", 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002904401 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283927+0000", 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00038193199999999997 2024-09-13T11:34:05.009 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283964+0000", 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7117999999999999e-05 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284070+0000", 2024-09-13T11:34:05.010 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010545899999999999 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:41 3.3 3:e9fa9a15:::3c65c6c9770b7a29cd5a9b6dc5b0b39ab23cb2a64bfb19442efcb911d255e067:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.670881836000007, 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013499547000000001, 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.011 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270866+0000", 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.012 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270867+0000", 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0900000000000001e-07 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270893+0000", 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6057000000000001e-05 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270901+0000", 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0299999999999994e-06 2024-09-13T11:34:05.013 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270915+0000", 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4450000000000001e-05 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271104+0000", 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018904600000000001 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271111+0000", 2024-09-13T11:34:05.014 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6039999999999998e-06 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280674+0000", 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0095629140000000005 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.280811+0000", 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000136802 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.015 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281027+0000", 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000216617 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283802+0000", 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027746540000000001 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283884+0000", 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2167000000000001e-05 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.016 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284178+0000", 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029347899999999997 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284246+0000", 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8609999999999995e-05 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284366+0000", 2024-09-13T11:34:05.017 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011950799999999999 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(osd.3.22:43 3.3 3:dc8816b1:::1795f5bdb26201908336f20eb1cf44426828990dc4449d47ce29af6145297222:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.670821141000005, 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013665096999999999, 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.018 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.019 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270927+0000", 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7999999999999995e-07 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270933+0000", 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2410000000000003e-06 2024-09-13T11:34:05.020 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270939+0000", 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5810000000000002e-06 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.270945+0000", 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5659999999999997e-06 2024-09-13T11:34:05.021 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271134+0000", 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000189161 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.271140+0000", 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2999999999999998e-06 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.022 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281407+0000", 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010266226 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281549+0000", 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000142583 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.023 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 13,15", 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281737+0000", 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00018791500000000001 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283810+0000", 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002072571 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284127+0000", 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000317516 2024-09-13T11:34:05.024 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284417+0000", 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00029000899999999998 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284480+0000", 2024-09-13T11:34:05.025 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3136e-05 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284592+0000", 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111712 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2303 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:05.026 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 69.096993268999995, 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43316489499999999, 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.027 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2303 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.028 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844755+0000", 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844756+0000", 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0599999999999999e-07 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.029 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.030 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844760+0000", 2024-09-13T11:34:05.030 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.0550000000000001e-06 2024-09-13T11:34:05.030 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.030 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.030 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844769+0000", 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6129999999999993e-06 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844776+0000", 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0280000000000003e-06 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:34:05.031 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7639000000000002e-05 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844841+0000", 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.1049999999999997e-06 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277896+0000", 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43305493499999997 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.032 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277920+0000", 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3614000000000001e-05 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 69.096941635999997, 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43315226600000001, 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.033 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.034 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844806+0000", 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844807+0000", 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3200000000000003e-07 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844811+0000", 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.799e-06 2024-09-13T11:34:05.035 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844827+0000", 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5461999999999999e-05 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844834+0000", 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.126e-06 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.036 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5369999999999999e-05 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844863+0000", 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.658e-06 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277951+0000", 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308847900000003 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.037 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277959+0000", 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4399999999999999e-06 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4.70670c31 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 69.096899925000002, 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43314228399999999, 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.038 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "reached pg", 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.039 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844848+0000", 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844849+0000", 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0100000000000005e-07 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844853+0000", 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9589999999999997e-06 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.040 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844859+0000", 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4110000000000001e-06 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844864+0000", 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8930000000000004e-06 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.041 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844892+0000", 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8042e-05 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:55.844898+0000", 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.5049999999999998e-06 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277984+0000", 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.43308647700000003 2024-09-13T11:34:05.042 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:56.277990+0000", 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3960000000000003e-06 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2304 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 153896~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:05.043 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.659249012999993, 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014013433, 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2304 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.044 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282499+0000", 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282500+0000", 2024-09-13T11:34:05.045 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5499999999999998e-07 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282503+0000", 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.563e-06 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282510+0000", 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8820000000000003e-06 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.046 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282516+0000", 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2169999999999999e-06 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282562+0000", 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5868999999999997e-05 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282570+0000", 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6539999999999999e-06 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.047 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296085+0000", 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013515255 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296187+0000", 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000101457 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.048 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296512+0000", 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00032588099999999998 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2305 4.11 4:8c30e60e:::smithi20438004-18:head [sparse-read 447518~1761280] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e60)", 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 67.659164520000004, 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014496826000000001, 2024-09-13T11:34:05.049 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2305 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.050 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282584+0000", 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5000000000000002e-07 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282589+0000", 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4220000000000002e-06 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.051 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282595+0000", 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8789999999999997e-06 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282606+0000", 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0229e-05 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282652+0000", 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6610999999999998e-05 2024-09-13T11:34:05.052 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.282661+0000", 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2640000000000008e-06 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296550+0000", 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013888981999999999 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.296628+0000", 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.8564000000000002e-05 2024-09-13T11:34:05.053 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:57.297080+0000", 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000452225 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4497 4.1c 4:3a6b347b:::smithi20438004-49:head [read 39936~1781760] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e90)", 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:49.488729+0000", 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.453019532000001, 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017108786000000001, 2024-09-13T11:34:05.054 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4497 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488729+0000", 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.055 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488729+0000", 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488730+0000", 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9939999999999999e-06 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488736+0000", 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9859999999999999e-06 2024-09-13T11:34:05.056 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488752+0000", 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5872999999999999e-05 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.488764+0000", 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1165000000000001e-05 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.057 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.491323+0000", 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025592029999999999 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.491338+0000", 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4849e-05 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.499774+0000", 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0084368629999999993 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.058 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.499959+0000", 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000184737 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.505837+0000", 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0058781160000000001 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.059 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4499 4.1c 4:3a6b347b:::smithi20438004-49:head [assert-version v44,set-chunk] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:49.540567+0000", 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 15.401181166000001, 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015634990000000001, 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.060 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4499 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540567+0000", 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540567+0000", 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.061 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540568+0000", 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2100000000000001e-06 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540575+0000", 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2559999999999996e-06 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540585+0000", 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.2469999999999997e-06 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.062 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.540590+0000", 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7250000000000002e-06 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.542963+0000", 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.002372809 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.543053+0000", 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.0259000000000004e-05 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.063 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 4,6,10,15", 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.547232+0000", 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041787359999999997 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.547764+0000", 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00053142700000000001 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.547973+0000", 2024-09-13T11:34:05.064 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00020973500000000001 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.549921+0000", 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00194754 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.555923+0000", 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0060020309999999997 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.555976+0000", 2024-09-13T11:34:05.065 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3100999999999999e-05 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.556049+0000", 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2733999999999999e-05 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:49.556202+0000", 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015317999999999999 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.066 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.067 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.067 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:05.068 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-13T11:34:05.069 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-09-13T11:34:05.067+0000 7fc28faac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33335) UID: 0 2024-09-13T11:34:05.169 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-13T11:34:05.167+0000 7f8a8cfee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32870) UID: 0 2024-09-13T11:34:05.211 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:05.211 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:05.212 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:34:05.212 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:05.220 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-13T11:34:05.269 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-13T11:34:05.268+0000 7fb9b8b7f640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 32875) UID: 0 2024-09-13T11:34:05.281 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_0' removed 2024-09-13T11:34:05.281 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-09-13T11:34:05.302 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-13T11:34:05.313 INFO:tasks.thrashosds:joining thrashosds 2024-09-13T11:34:05.325 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:05.325 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:05.326 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:34:05.326 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:34:05.326 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:05.334 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-13T11:34:05.439 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:05.439 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:34:05.439 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:34:05.439 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:34:05.439 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.439 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:215 3.1 e38/30 3:84f0de12:::smithi20438004-16 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.336242057000007, 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016706373, 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.440 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101079+0000", 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.99e-07 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104537+0000", 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0034575700000000001 2024-09-13T11:34:05.441 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104552+0000", 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5475999999999999e-05 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104566+0000", 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3766e-05 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.442 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104623+0000", 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7046999999999998e-05 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.104650+0000", 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6786000000000001e-05 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117752+0000", 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013102387 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.443 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.117785+0000", 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.2741999999999999e-05 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:210 3.1 e38/30 3:85041882:::smithi20438004-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'4, mlcod=0'0)", 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.332282731000006, 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025912691000000002, 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.444 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105038+0000", 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.445 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.105039+0000", 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7100000000000001e-07 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108050+0000", 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030115910000000001 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108061+0000", 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0512e-05 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.446 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108069+0000", 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.4479999999999994e-06 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108126+0000", 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7024999999999997e-05 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108165+0000", 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8701000000000001e-05 2024-09-13T11:34:05.447 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130916+0000", 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022750829 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130951+0000", 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.4913999999999998e-05 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.448 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:203 3.1 e38/30 3:970ba768:::smithi20438004-4 low_tier:head v 38'5, mlcod=0'0)", 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.329076237999999, 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.031827052000000002, 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.449 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108244+0000", 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.108245+0000", 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1800000000000005e-07 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111202+0000", 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0029571440000000001 2024-09-13T11:34:05.450 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111221+0000", 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8887000000000001e-05 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111248+0000", 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6458e-05 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111707+0000", 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00045935799999999999 2024-09-13T11:34:05.451 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111729+0000", 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2146000000000001e-05 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140022+0000", 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028292571999999998 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.452 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140071+0000", 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9669000000000002e-05 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:202 3.1 e38/30 3:830e84e2:::smithi20438004-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'6, mlcod=0'0)", 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.326052688000004, 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028930938999999999, 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.453 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111268+0000", 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.111269+0000", 2024-09-13T11:34:05.454 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5700000000000002e-07 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113854+0000", 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025854340000000002 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113866+0000", 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1884e-05 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.455 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.113873+0000", 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.8530000000000003e-06 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116175+0000", 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0023019 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.116218+0000", 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.2796000000000002e-05 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.456 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140165+0000", 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023947718999999999 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.140199+0000", 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3596000000000002e-05 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.457 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:218 3.1 e38/30 3:8b21ee9a:::smithi20438004-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'7, mlcod=38'2)", 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.316189780000002, 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025863384999999999, 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.458 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121131+0000", 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.121132+0000", 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3260000000000001e-06 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124734+0000", 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0036019379999999998 2024-09-13T11:34:05.459 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124748+0000", 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3913999999999999e-05 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124760+0000", 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1786e-05 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.460 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124849+0000", 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8882999999999996e-05 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.124874+0000", 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5466999999999999e-05 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146945+0000", 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022070837999999999 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.461 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146994+0000", 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9233000000000001e-05 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:221 3.1 e38/30 3:baf08732:::smithi20438004-22 low_tier:head v 38'8, mlcod=38'2)", 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.309256597000001, 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020872166000000001, 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.462 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128064+0000", 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128066+0000", 2024-09-13T11:34:05.463 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.86e-06 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131456+0000", 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0033901360000000002 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131467+0000", 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0806e-05 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.464 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131474+0000", 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.627e-06 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131524+0000", 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9262999999999999e-05 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131554+0000", 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0148999999999999e-05 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.465 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148883+0000", 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017329445999999998 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.148936+0000", 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2879e-05 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:219 3.1 e38/30 3:97e533d4:::smithi20438004-20 low_tier:head v 38'9, mlcod=38'2)", 2024-09-13T11:34:05.466 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.305818398, 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017580887, 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.467 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131502+0000", 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.131503+0000", 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7100000000000002e-07 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.133995+0000", 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0024920630000000001 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.468 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134006+0000", 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0696000000000001e-05 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.134012+0000", 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4189999999999999e-06 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136541+0000", 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0025286150000000001 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.469 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.136563+0000", 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2531000000000002e-05 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149031+0000", 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012467951 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.149083+0000", 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2040999999999999e-05 2024-09-13T11:34:05.470 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:231 3.1 e38/30 3:ab075b6a:::smithi20438004-32 low_tier:head v 38'10, mlcod=38'7)", 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.279609820000005, 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022274233000000001, 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.471 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157711+0000", 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.157712+0000", 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3689999999999999e-06 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.472 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161974+0000", 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0042619160000000001 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161990+0000", 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5506e-05 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161998+0000", 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.7520000000000002e-06 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.473 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162103+0000", 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000104415 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162126+0000", 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3643e-05 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179932+0000", 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017805476000000001 2024-09-13T11:34:05.474 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.179985+0000", 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3155999999999997e-05 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:235 3.1 e38/30 3:be4a0686:::smithi20438004-36 low_tier:head v 38'11, mlcod=38'7)", 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.275284587000002, 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018684003000000001, 2024-09-13T11:34:05.475 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162036+0000", 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.476 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.162037+0000", 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.083e-06 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166161+0000", 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0041238799999999999 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166179+0000", 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8329999999999999e-05 2024-09-13T11:34:05.477 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166190+0000", 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0822000000000001e-05 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166302+0000", 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00011233700000000001 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166328+0000", 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5185999999999999e-05 2024-09-13T11:34:05.478 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180646+0000", 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014318261000000001 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180720+0000", 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4103999999999994e-05 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.479 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:230 3.1 e38/30 3:970d9811:::smithi20438004-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'12, mlcod=38'8)", 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.271105585000001, 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014657148, 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.480 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.166215+0000", 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1300000000000001e-07 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169754+0000", 2024-09-13T11:34:05.481 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0035389779999999999 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169772+0000", 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8020999999999999e-05 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169782+0000", 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0037999999999999e-05 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.482 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169847+0000", 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4695000000000003e-05 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.169865+0000", 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.7365999999999999e-05 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180834+0000", 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010969016 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.483 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.180872+0000", 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.8621000000000002e-05 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:237 3.1 e38/30 3:9a2730a7:::smithi20438004-38 low_tier:head v 38'13, mlcod=38'8)", 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.259194432000001, 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018123856000000001, 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.484 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178126+0000", 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.485 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178128+0000", 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4309999999999999e-06 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181360+0000", 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0032324699999999999 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181371+0000", 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0978000000000001e-05 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.486 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181378+0000", 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.861e-06 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181431+0000", 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3173000000000001e-05 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181455+0000", 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.4236999999999999e-05 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.487 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196213+0000", 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01475807 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.196250+0000", 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.6636e-05 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.488 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:236 3.1 e38/30 3:97ae5972:::smithi20438004-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'14, mlcod=38'9)", 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.255910874999998, 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016397140000000001, 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.489 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.492 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.492 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:34:05.492 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.492 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.492 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181410+0000", 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.181411+0000", 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7399999999999991e-07 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184468+0000", 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0030575759999999998 2024-09-13T11:34:05.493 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184481+0000", 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3072e-05 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184491+0000", 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0068000000000001e-05 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184550+0000", 2024-09-13T11:34:05.494 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8251999999999997e-05 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184575+0000", 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5689000000000001e-05 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197720+0000", 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013144708999999999 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197807+0000", 2024-09-13T11:34:05.495 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6799999999999996e-05 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:281 4.a 4:56ef4fd4:::smithi20438004-32:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.204872163000005, 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018358335999999999, 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.496 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 281 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.497 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232448+0000", 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232449+0000", 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.7499999999999998e-07 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232457+0000", 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3610000000000003e-06 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232470+0000", 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3230999999999999e-05 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232478+0000", 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5989999999999996e-06 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232541+0000", 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3717e-05 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.499 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.232619+0000", 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7996000000000003e-05 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 9,10", 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248106+0000", 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015486616 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.248991+0000", 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00088509199999999998 2024-09-13T11:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249363+0000", 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00037176599999999999 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250719+0000", 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001356255 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.501 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250765+0000", 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6053000000000001e-05 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250807+0000", 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1675000000000003e-05 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1696 4.b 4:da92e835:::smithi20438004-50:head [tier-promote] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:34:05.502 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 79.168720347999994, 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016782392, 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1696 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:34:05.503 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268600+0000", 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268602+0000", 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6160000000000001e-06 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.504 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268608+0000", 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7629999999999997e-06 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268620+0000", 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1950000000000001e-05 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268632+0000", 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2330999999999999e-05 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.505 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269210+0000", 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00057761900000000003 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269221+0000", 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.168e-05 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277483+0000", 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0082621180000000006 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.506 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.277641+0000", 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000157365 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285383+0000", 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00774195 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.507 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1726 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 79.168651890999996, 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015213417999999999, 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1726 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.508 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268669+0000", 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.58e-07 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.509 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268673+0000", 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7809999999999999e-06 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268679+0000", 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7729999999999996e-06 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268684+0000", 2024-09-13T11:34:05.510 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1950000000000002e-06 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269275+0000", 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059124500000000001 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269281+0000", 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2779999999999999e-06 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281010+0000", 2024-09-13T11:34:05.511 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011728762 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.281114+0000", 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000103872 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.283882+0000", 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027680539999999998 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.512 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1727 4.b 4:de617177:::smithi20438004-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 79.168630144000005, 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016698689999999999, 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1727 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.513 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268690+0000", 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268691+0000", 2024-09-13T11:34:05.514 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6100000000000001e-07 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268694+0000", 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.672e-06 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268699+0000", 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1819999999999997e-06 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.515 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.268703+0000", 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269302+0000", 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00059838199999999995 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.269307+0000", 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3170000000000001e-06 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.516 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284218+0000", 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014911308999999999 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.284330+0000", 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000111715 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:46.285389+0000", 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0010592729999999999 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.517 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2018 4.5 4:a9b0c9bd:::smithi20438004-37:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e54)", 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 74.561662467000005, 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.021048396, 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.518 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2018 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875658+0000", 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.519 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875659+0000", 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.9000000000000002e-07 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875661+0000", 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.9920000000000002e-06 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875665+0000", 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5210000000000004e-06 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.520 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875668+0000", 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3220000000000001e-06 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875714+0000", 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5189999999999999e-05 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.875770+0000", 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6824999999999999e-05 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.521 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:50.896707+0000", 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020936156000000001 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3496 4.18 4:1cf41451:::smithi20438004-7:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 41.683296544000001, 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018591032, 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.522 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3496 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.523 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754024+0000", 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754026+0000", 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4610000000000001e-06 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754033+0000", 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.0369999999999997e-06 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754047+0000", 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4063000000000001e-05 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754056+0000", 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5980000000000003e-06 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754109+0000", 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2599999999999998e-05 2024-09-13T11:34:05.525 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.754165+0000", 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6079999999999998e-05 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:23.772615+0000", 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018450193 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.526 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4005 4.a 4:56ef4fd4:::smithi20438004-32:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e81)", 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 30.757648179, 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018951355999999999, 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4005 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.527 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679672+0000", 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679675+0000", 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2079999999999999e-06 2024-09-13T11:34:05.528 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679682+0000", 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3839999999999999e-06 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679701+0000", 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865999999999999e-05 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679710+0000", 2024-09-13T11:34:05.529 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.1980000000000007e-06 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679782+0000", 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.2163999999999996e-05 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.679869+0000", 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6366000000000006e-05 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.530 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:34.698624+0000", 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018755170000000002 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:4055 4.b 4:d738a69d:::smithi20438004-16:head [write 400099~616131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 29.839139709000001, 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01451794, 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.531 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 4055 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598181+0000", 2024-09-13T11:34:05.532 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598185+0000", 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.4120000000000003e-06 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598800+0000", 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00061453600000000005 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598830+0000", 2024-09-13T11:34:05.533 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.0593999999999999e-05 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598847+0000", 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6552e-05 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.598988+0000", 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00014121000000000001 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op must wait for map", 2024-09-13T11:34:05.534 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.599001+0000", 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2605e-05 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603641+0000", 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0046401250000000002 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603745+0000", 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00010398700000000001 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.535 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 0,4", 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.603921+0000", 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00017563 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.607101+0000", 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0031806360000000001 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.611101+0000", 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0039995650000000001 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.536 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612503+0000", 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0014018909999999999 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612653+0000", 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00015040499999999999 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:35.612699+0000", 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.5791999999999998e-05 2024-09-13T11:34:05.537 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.538 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.538 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.538 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.538 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.538 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:05.539 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-13T11:34:05.642 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:05.643 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:05.643 INFO:teuthology.orchestra.run.smithi204.stdout: "num_ops": 0 2024-09-13T11:34:05.643 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:05.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-13T11:34:05.758 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:05.758 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [], 2024-09-13T11:34:05.759 INFO:teuthology.orchestra.run.smithi204.stdout: "complaint_time": 30, 2024-09-13T11:34:05.759 INFO:teuthology.orchestra.run.smithi204.stdout: "num_blocked_ops": 0 2024-09-13T11:34:05.759 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:05.767 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout:{ 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "size": 20, 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 600, 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "ops": [ 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:204 3.0 e38/30 3:34322f4c:::smithi20438004-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'1, mlcod=0'0)", 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.778584598999998, 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017691972, 2024-09-13T11:34:05.872 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091378+0000", 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.873 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.091380+0000", 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2899999999999999e-06 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101598+0000", 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010218677000000001 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101615+0000", 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6441999999999999e-05 2024-09-13T11:34:05.874 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101623+0000", 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.6200000000000005e-06 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101689+0000", 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5165000000000004e-05 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101711+0000", 2024-09-13T11:34:05.875 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2065e-05 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109006+0000", 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0072957489999999998 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.109070+0000", 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.3963999999999999e-05 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.876 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:205 3.0 e38/30 3:03052d75:::smithi20438004-6 low_tier:head v 38'2, mlcod=0'0)", 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.768310032000002, 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028764082999999999, 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.877 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101653+0000", 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.101654+0000", 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.3799999999999996e-07 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114256+0000", 2024-09-13T11:34:05.878 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012602402 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114281+0000", 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5181000000000002e-05 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114294+0000", 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2347999999999999e-05 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.879 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114356+0000", 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.2073e-05 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114376+0000", 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0656999999999999e-05 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130336+0000", 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.015960164999999998 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.880 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.130417+0000", 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0519000000000006e-05 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:213 3.0 e38/30 3:22c1b591:::smithi20438004-14 low_tier:head v 38'3, mlcod=0'0)", 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.755642131000002, 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025024916000000001, 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.881 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.114321+0000", 2024-09-13T11:34:05.882 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7199999999999999e-07 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119935+0000", 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0056134139999999997 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119946+0000", 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1653e-05 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.119954+0000", 2024-09-13T11:34:05.883 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.9100000000000005e-06 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120019+0000", 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4399999999999993e-05 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120044+0000", 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.5684999999999999e-05 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.884 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139292+0000", 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.019248000000000001 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.139346+0000", 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3282000000000002e-05 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:217 3.0 e38/30 3:33b1e45a:::smithi20438004-18 low_tier:head v 38'4, mlcod=0'0)", 2024-09-13T11:34:05.885 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.749032283999995, 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.025257208, 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.886 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120931+0000", 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.120932+0000", 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6649999999999999e-06 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128546+0000", 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007613982 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.887 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128559+0000", 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3073e-05 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128571+0000", 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1739000000000001e-05 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128694+0000", 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00012279899999999999 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.888 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.128723+0000", 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.8568000000000002e-05 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146136+0000", 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.017413537999999999 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.146188+0000", 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1844e-05 2024-09-13T11:34:05.889 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:222 3.0 e38/30 3:3a2d9873:::smithi20438004-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'5, mlcod=0'0)", 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.740655254000004, 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.018364128, 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.890 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129308+0000", 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.129309+0000", 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.401e-06 2024-09-13T11:34:05.891 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135276+0000", 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059666140000000003 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135288+0000", 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1977000000000001e-05 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.892 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135295+0000", 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.9630000000000001e-06 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135347+0000", 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2716999999999997e-05 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.135369+0000", 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1642e-05 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.893 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147629+0000", 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012259535 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.147672+0000", 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3278999999999997e-05 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:227 3.0 e38/30 3:162d2d40:::smithi20438004-28 low_tier:head v 38'7, mlcod=38'3)", 2024-09-13T11:34:05.894 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.715524075000005, 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012944713, 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.895 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.154439+0000", 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.44e-07 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161247+0000", 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0068071579999999998 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.896 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161260+0000", 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2995e-05 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161268+0000", 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1699999999999997e-06 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.897 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161326+0000", 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.8046e-05 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.161347+0000", 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1239000000000001e-05 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167320+0000", 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0059725059999999998 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.898 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.167384+0000", 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4054999999999996e-05 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:232 3.0 e38/30 3:2c6543fb:::smithi20438004-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'8, mlcod=38'5)", 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:34:05.899 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.706622796999994, 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014990427000000001, 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163340+0000", 2024-09-13T11:34:05.900 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.163342+0000", 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.742e-06 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170876+0000", 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0075341289999999997 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.901 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170898+0000", 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2299000000000001e-05 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170908+0000", 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.713e-06 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171007+0000", 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.8472000000000002e-05 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.902 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.171044+0000", 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.7527999999999997e-05 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178279+0000", 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007234847 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.178331+0000", 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.1697000000000003e-05 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.903 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:234 3.0 e38/30 3:2cb9a734:::smithi20438004-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 38'9, mlcod=38'5)", 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.699025074000005, 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013627746, 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.904 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170938+0000", 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.170939+0000", 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0410000000000001e-06 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.905 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175690+0000", 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0047511749999999998 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175708+0000", 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8235000000000001e-05 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175717+0000", 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.8969999999999997e-06 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.906 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175784+0000", 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6601999999999997e-05 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.175804+0000", 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0491000000000001e-05 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184509+0000", 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087043069999999997 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.907 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.184566+0000", 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.6997999999999999e-05 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:239 3.0 e38/30 3:3c12287e:::smithi20438004-40 low_tier:head v 38'10, mlcod=38'6)", 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.693621144999994, 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.020844166000000001, 2024-09-13T11:34:05.908 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176342+0000", 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.909 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.176343+0000", 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.249e-06 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188938+0000", 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012594453 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188947+0000", 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.3270000000000007e-06 2024-09-13T11:34:05.910 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188952+0000", 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9810000000000003e-06 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188999+0000", 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7441000000000002e-05 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.189014+0000", 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.5051999999999999e-05 2024-09-13T11:34:05.911 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197137+0000", 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0081225250000000002 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.197186+0000", 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9138000000000003e-05 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.912 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:241 3.0 e38/30 3:2c9c5010:::smithi20438004-42 low_tier:head v 38'11, mlcod=38'6)", 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.680997454999996, 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.023047494000000002, 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:34:05.913 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.188966+0000", 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3300000000000003e-07 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.914 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203102+0000", 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.014135867999999999 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203116+0000", 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3917000000000001e-05 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203122+0000", 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.6610000000000003e-06 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.915 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203180+0000", 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7951999999999997e-05 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203202+0000", 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2098000000000001e-05 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211949+0000", 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0087465400000000006 2024-09-13T11:34:05.916 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212013+0000", 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.4024999999999998e-05 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(client.4631.0:243 3.0 e38/30 3:03808b00:::smithi20438004-44 low_tier:head v 38'12, mlcod=38'8)", 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.666818004999996, 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016174146, 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.917 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.918 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.203145+0000", 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.3000000000000001e-07 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212419+0000", 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0092735720000000008 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212441+0000", 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.2212999999999999e-05 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.919 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212455+0000", 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3706000000000001e-05 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212523+0000", 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7798000000000004e-05 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.212544+0000", 2024-09-13T11:34:05.920 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.1222000000000001e-05 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219265+0000", 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.006720625 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.219319+0000", 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.448e-05 2024-09-13T11:34:05.921 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:255 4.e 4:7022fc62:::smithi20438004-6:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.658599836999997, 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.011905948, 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.922 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 255 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211363+0000", 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.923 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211365+0000", 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.0269999999999998e-06 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211378+0000", 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2948e-05 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211401+0000", 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.3353000000000002e-05 2024-09-13T11:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211413+0000", 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.1535000000000001e-05 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211496+0000", 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.3330000000000003e-05 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.211605+0000", 2024-09-13T11:34:05.925 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000108972 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.220209+0000", 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0086035980000000005 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.221566+0000", 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.001357228 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.926 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.222278+0000", 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00071161 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223173+0000", 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00089538900000000002 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223222+0000", 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8952999999999997e-05 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.927 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.223269+0000", 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.7005000000000001e-05 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:297 4.e 4:727deab8:::smithi20438004-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:05.928 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.627776433999998, 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.01104336, 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 297 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:34:05.929 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242187+0000", 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6799999999999999e-07 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.930 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242191+0000", 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1919999999999998e-06 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242201+0000", 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.6649999999999998e-06 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242206+0000", 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.2850000000000002e-06 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.931 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242255+0000", 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.8139000000000001e-05 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.242336+0000", 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1557999999999994e-05 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for subops from 5,14", 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.249076+0000", 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067403439999999997 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.932 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250046+0000", 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00096943099999999998 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "sub_op_commit_rec", 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.250410+0000", 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00036377099999999998 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "op_commit", 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253155+0000", 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0027453149999999999 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.933 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253197+0000", 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.1659000000000002e-05 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.253230+0000", 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3232999999999998e-05 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.934 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:787 4.e 4:7022fc62:::smithi20438004-6:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 98.481061647000004, 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016670758000000001, 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 787 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.935 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388901+0000", 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.936 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388903+0000", 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.6419999999999999e-06 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388911+0000", 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.2919999999999996e-06 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388926+0000", 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4543e-05 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.937 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.388935+0000", 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5829999999999996e-06 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389001+0000", 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.5513999999999996e-05 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.389096+0000", 2024-09-13T11:34:05.938 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 9.5118999999999995e-05 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:27.405572+0000", 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.016476065000000002 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1393 4.c 4:3a6b347b:::smithi20438004-49:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e43)", 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 89.775889594999995, 2024-09-13T11:34:05.939 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028348306, 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1393 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.940 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094073+0000", 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094074+0000", 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.096e-06 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094079+0000", 2024-09-13T11:34:05.941 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.3789999999999999e-06 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094089+0000", 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.0298e-05 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094097+0000", 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.7670000000000003e-06 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094157+0000", 2024-09-13T11:34:05.942 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.0556000000000001e-05 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.094265+0000", 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000107346 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.122422+0000", 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.028156864 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.943 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_repop(osd.1.0:180 4.1 e44/31 4:84fa1f64:::smithi20438004-43:head v 44'71, mlcod=43'69)", 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 89.027937225000002, 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013151014000000001, 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:34:05.944 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842026+0000", 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.842045+0000", 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8865e-05 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.945 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848182+0000", 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0061372370000000002 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848205+0000", 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.298e-05 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848218+0000", 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.348e-05 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.946 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848300+0000", 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.1766000000000003e-05 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.848327+0000", 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.6446999999999999e-05 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "commit_sent", 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855119+0000", 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0067926490000000004 2024-09-13T11:34:05.947 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:36.855177+0000", 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.7590000000000003e-05 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1458 4.e 4:7984cf84:::smithi20438004-46:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 88.842325173000006, 2024-09-13T11:34:05.948 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012793834, 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1458 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.949 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027638+0000", 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027640+0000", 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.046e-06 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.950 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027651+0000", 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.102e-05 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027669+0000", 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.8182999999999999e-05 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027681+0000", 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.2177999999999999e-05 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.951 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027762+0000", 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 8.0331e-05 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.027868+0000", 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.000106443 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:37.040432+0000", 2024-09-13T11:34:05.952 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.012563632999999999 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:1787 4.e 4:7022fc62:::smithi20438004-6:head [read 188928~1265152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 78.596232482000005, 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.013752110999999999, 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.953 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 1787 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.954 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273730+0000", 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273731+0000", 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.9900000000000001e-07 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273734+0000", 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 3.3510000000000002e-06 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.955 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273741+0000", 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.7109999999999998e-06 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.273787+0000", 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 4.6198000000000001e-05 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.956 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274432+0000", 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00064478300000000001 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "waiting for peered", 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.274461+0000", 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 2.9051999999999999e-05 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281751+0000", 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.007289963 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.957 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.281984+0000", 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.00023257099999999999 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:32:47.287483+0000", 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.0054989829999999998 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.958 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:2762 4.e 4:7984cf84:::smithi20438004-46:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e66)", 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 58.973775064000002, 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022335361000000001, 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 2762 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.959 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896188+0000", 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896189+0000", 2024-09-13T11:34:05.960 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3519999999999999e-06 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896196+0000", 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.81e-06 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896210+0000", 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.3439e-05 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.961 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896217+0000", 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.5889999999999997e-06 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896278+0000", 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1072000000000001e-05 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.962 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.896338+0000", 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9998999999999999e-05 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:06.918523+0000", 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.022185099999999999 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.963 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: "description": "osd_op(client.4631.0:3763 4.1 4:8533c921:::smithi20438004-10:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: "initiated_at": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: "age": 36.736220062999998, 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010966191, 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: "continuous": false, 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: "type_data": { 2024-09-13T11:34:05.964 INFO:teuthology.orchestra.run.smithi204.stdout: "flag_point": "started", 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "client_info": { 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "client": "client.4631", 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "client_addr": "172.21.15.204:0/1764892635", 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "tid": 3763 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "events": [ 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "initiated", 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "header_read", 2024-09-13T11:34:05.965 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133743+0000", 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "throttled", 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133745+0000", 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.75e-06 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "all_read", 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133752+0000", 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.6070000000000002e-06 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "dispatched", 2024-09-13T11:34:05.966 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133767+0000", 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 1.4577e-05 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "queued_for_pg", 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133774+0000", 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 7.4159999999999998e-06 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "reached_pg", 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133836+0000", 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 6.1787999999999999e-05 2024-09-13T11:34:05.967 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "started", 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.133896+0000", 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 5.9768000000000001e-05 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: }, 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: { 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: "event": "done", 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: "time": "2024-09-13T11:33:29.144709+0000", 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: "duration": 0.010813285000000001 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.968 INFO:teuthology.orchestra.run.smithi204.stdout: } 2024-09-13T11:34:05.969 INFO:teuthology.orchestra.run.smithi204.stdout: ] 2024-09-13T11:34:05.969 INFO:teuthology.orchestra.run.smithi204.stdout:} 2024-09-13T11:34:07.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-13T11:34:07.676 INFO:tasks.thrashosds.thrasher:reviving osd 2024-09-13T11:34:07.676 INFO:tasks.thrashosds.thrasher:Reviving osd 12 2024-09-13T11:34:07.677 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-13T11:34:07.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-13T11:34:07.680 INFO:tasks.ceph.osd.12:Started 2024-09-13T11:34:07.680 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:34:07.781 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:config diff 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:config diff get 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:config get 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:config help [] 2024-09-13T11:34:07.782 INFO:teuthology.orchestra.run.smithi038.stderr:config set ... 2024-09-13T11:34:07.783 INFO:teuthology.orchestra.run.smithi038.stderr:admin_socket: invalid command 2024-09-13T11:34:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:34:07.784 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['dump_ops_in_flight'] 2024-09-13T11:34:08.013 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-09-13T11:34:08.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-13T11:34:08.625 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:34:08.624+0000 7f6251902640 -1 Falling back to public interface 2024-09-13T11:34:09.025 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-09-13T11:34:12.391 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:34:12.390+0000 7f6251902640 -1 osd.12 58 log_to_monitors true 2024-09-13T11:34:12.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator fragmentation block 2024-09-13T11:34:12.902 INFO:teuthology.orchestra.run.smithi038.stderr:admin_socket: invalid command 2024-09-13T11:34:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:34:12.905 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-12, ['dump_ops_in_flight'] 2024-09-13T11:34:13.340 INFO:tasks.ceph.osd.12.smithi038.stderr:2024-09-13T11:34:13.337+0000 7f62496b8640 -1 osd.12 107 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:34:17.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-13T11:34:18.027 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:34:18.027 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-09-13T11:34:18.028 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-09-13T11:34:18.028 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:34:18.036 INFO:tasks.thrashosds.thrasher:reviving osd 2024-09-13T11:34:18.036 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-13T11:34:18.036 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T11:34:18.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T11:34:18.039 INFO:tasks.ceph.osd.2:Started 2024-09-13T11:34:18.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:34:18.149 INFO:teuthology.orchestra.run.smithi154.stderr:no valid command found; 10 closest matches: 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:0 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:1 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:2 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:abort 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:assert 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:config diff 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:config diff get 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:config get 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:config help [] 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:config set ... 2024-09-13T11:34:18.150 INFO:teuthology.orchestra.run.smithi154.stderr:admin_socket: invalid command 2024-09-13T11:34:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:34:18.152 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-13T11:34:19.466 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:34:19.464+0000 7fdbdec11640 -1 Falling back to public interface 2024-09-13T11:34:23.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:34:23.255 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:34:23.253+0000 7fdbdec11640 -1 osd.2 39 log_to_monitors true 2024-09-13T11:34:23.268 INFO:teuthology.orchestra.run.smithi154.stderr:no valid command found; 10 closest matches: 2024-09-13T11:34:23.268 INFO:teuthology.orchestra.run.smithi154.stderr:0 2024-09-13T11:34:23.268 INFO:teuthology.orchestra.run.smithi154.stderr:1 2024-09-13T11:34:23.268 INFO:teuthology.orchestra.run.smithi154.stderr:2 2024-09-13T11:34:23.268 INFO:teuthology.orchestra.run.smithi154.stderr:abort 2024-09-13T11:34:23.268 INFO:teuthology.orchestra.run.smithi154.stderr:assert 2024-09-13T11:34:23.269 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs debug_inject_read_zeros 2024-09-13T11:34:23.269 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs files list 2024-09-13T11:34:23.269 INFO:teuthology.orchestra.run.smithi154.stderr:bluefs stats 2024-09-13T11:34:23.269 INFO:teuthology.orchestra.run.smithi154.stderr:bluestore allocator dump block 2024-09-13T11:34:23.269 INFO:teuthology.orchestra.run.smithi154.stderr:bluestore allocator fragmentation block 2024-09-13T11:34:23.269 INFO:teuthology.orchestra.run.smithi154.stderr:admin_socket: invalid command 2024-09-13T11:34:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T11:34:23.270 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-13T11:34:24.146 INFO:tasks.ceph.osd.2.smithi154.stderr:2024-09-13T11:34:24.144+0000 7fdbd23b1640 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T11:34:28.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-13T11:34:28.389 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-13T11:34:28.389 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-13T11:34:28.389 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-13T11:34:28.389 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-13T11:34:28.398 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T11:34:28.398 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-13T11:34:28.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-09-13T11:34:28.722 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:34:28.720+0000 7f8352d34640 -1 mon.a@0(leader).osd e119 definitely_dead 0 2024-09-13T11:34:29.137 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.5. 2024-09-13T11:34:29.152 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-13T11:34:29.152 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T11:34:29.153 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-13T11:34:29.153 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-13T11:34:29.464 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:34:29.463+0000 7f8352d34640 -1 mon.a@0(leader).osd e120 definitely_dead 0 2024-09-13T11:34:30.139 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.1. 2024-09-13T11:34:30.157 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-13T11:34:30.157 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-13T11:34:30.157 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-09-13T11:34:30.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 9 2024-09-13T11:34:30.472 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-09-13T11:34:30.470+0000 7f8352d34640 -1 mon.a@0(leader).osd e122 definitely_dead 0 2024-09-13T11:34:31.402 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.9. 2024-09-13T11:34:31.418 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-09-13T11:34:31.419 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-13T11:34:31.540 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-09-13T11:34:31.541 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:34:31.550 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-09-13T11:34:31.550 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:34:31.559 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-09-13T11:34:31.560 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-13T11:34:31.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-13T11:34:32.404 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-09-13T11:34:32.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-13T11:34:33.414 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-13T11:34:33.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-13T11:34:34.416 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-09-13T11:34:34.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-13T11:34:35.418 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-13T11:34:35.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-13T11:34:36.433 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.8 to 1 (10000) 2024-09-13T11:34:36.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-13T11:34:37.440 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-13T11:34:37.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-13T11:34:38.450 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-09-13T11:34:38.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-13T11:34:39.469 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-13T11:34:39.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-13T11:34:40.289 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-09-13T11:34:40.306 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-13T11:34:41.481 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-13T11:34:41.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-13T11:34:42.483 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.9 to 1 (10000) 2024-09-13T11:34:42.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-13T11:34:43.492 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-13T11:34:43.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-13T11:34:44.503 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.13 to 1 (10000) 2024-09-13T11:34:44.521 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-13T11:34:45.517 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-13T11:34:45.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-13T11:34:46.518 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-09-13T11:34:46.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-13T11:34:47.528 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-13T11:34:47.545 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-13T11:34:48.530 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.10 to 1 (10000) 2024-09-13T11:34:48.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-13T11:34:49.538 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-13T11:34:49.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-13T11:34:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.14 to 1 (10000) 2024-09-13T11:34:50.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-13T11:34:51.552 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-13T11:34:51.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-13T11:34:52.554 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-09-13T11:34:52.572 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-13T11:34:53.565 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-13T11:34:53.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-13T11:34:54.567 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-09-13T11:34:54.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-13T11:34:55.573 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-13T11:34:55.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-13T11:34:56.580 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.11 to 1 (10000) 2024-09-13T11:34:56.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-13T11:34:57.587 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-13T11:34:57.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-13T11:34:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.15 to 1 (10000) 2024-09-13T11:34:58.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-13T11:34:59.615 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-13T11:34:59.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-13T11:35:00.293 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.12 to 1 (10000) 2024-09-13T11:35:00.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-13T11:35:01.294 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-13T11:35:01.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-13T11:35:02.296 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-09-13T11:35:02.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-13T11:35:03.298 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-13T11:35:03.315 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-13T11:35:03.316 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-13T11:35:03.316 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-13T11:35:03.316 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-13T11:35:03.316 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T11:35:03.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:35:03.624 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:03.624 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":155,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:35:03.295526+0000","last_up_change":"2024-09-13T11:34:25.125921+0000","last_in_change":"2024-09-13T11:34:31.396117+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:32:02.947125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:32:05.120118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T11:32:17.683275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":7.940000057220459,"score_stable":7.940000057220459,"optimal_score":0.62999999523162842,"raw_score_acting":5,"raw_score_stable":5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2677506544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2677506544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2677506544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":2677506544}]},"public_addr":"172.21.15.38:6805/2677506544","cluster_addr":"172.21.15.38:6806/2677506544","heartbeat_back_addr":"172.21.15.38:6808/2677506544","heartbeat_front_addr":"172.21.15.38:6807/2677506544","state":["exists","up"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":3561806747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":3561806747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":3561806747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":3561806747}]},"public_addr":"172.21.15.118:6808/3561806747","cluster_addr":"172.21.15.118:6809/3561806747","heartbeat_back_addr":"172.21.15.118:6811/3561806747","heartbeat_front_addr":"172.21.15.118:6810/3561806747","state":["exists","up"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":39,"up_from":116,"up_thru":0,"down_at":41,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3302524927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":3302524927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":3302524927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3302524927}]},"public_addr":"172.21.15.154:6800/3302524927","cluster_addr":"172.21.15.154:6801/3302524927","heartbeat_back_addr":"172.21.15.154:6803/3302524927","heartbeat_front_addr":"172.21.15.154:6802/3302524927","state":["exists","up"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2326505959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":2326505959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":2326505959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2326505959}]},"public_addr":"172.21.15.204:6804/2326505959","cluster_addr":"172.21.15.204:6805/2326505959","heartbeat_back_addr":"172.21.15.204:6807/2326505959","heartbeat_front_addr":"172.21.15.204:6806/2326505959","state":["exists","up"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":3107658120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3107658120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3107658120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":3107658120}]},"public_addr":"172.21.15.38:6801/3107658120","cluster_addr":"172.21.15.38:6802/3107658120","heartbeat_back_addr":"172.21.15.38:6804/3107658120","heartbeat_front_addr":"172.21.15.38:6803/3107658120","state":["exists","up"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3800782738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":3800782738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":3800782738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":3800782738}]},"public_addr":"172.21.15.118:6800/3800782738","cluster_addr":"172.21.15.118:6801/3800782738","heartbeat_back_addr":"172.21.15.118:6803/3800782738","heartbeat_front_addr":"172.21.15.118:6802/3800782738","state":["exists","up"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3986362716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3986362716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3986362716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3986362716}]},"public_addr":"172.21.15.154:6808/3986362716","cluster_addr":"172.21.15.154:6809/3986362716","heartbeat_back_addr":"172.21.15.154:6811/3986362716","heartbeat_front_addr":"172.21.15.154:6810/3986362716","state":["exists","up"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1978982202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":1978982202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":1978982202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1978982202}]},"public_addr":"172.21.15.204:6808/1978982202","cluster_addr":"172.21.15.204:6809/1978982202","heartbeat_back_addr":"172.21.15.204:6811/1978982202","heartbeat_front_addr":"172.21.15.204:6810/1978982202","state":["exists","up"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3608171775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3608171775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3608171775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3608171775}]},"public_addr":"172.21.15.38:6813/3608171775","cluster_addr":"172.21.15.38:6814/3608171775","heartbeat_back_addr":"172.21.15.38:6816/3608171775","heartbeat_front_addr":"172.21.15.38:6815/3608171775","state":["exists","up"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2580659844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2580659844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":2580659844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2580659844}]},"public_addr":"172.21.15.118:6804/2580659844","cluster_addr":"172.21.15.118:6805/2580659844","heartbeat_back_addr":"172.21.15.118:6807/2580659844","heartbeat_front_addr":"172.21.15.118:6806/2580659844","state":["exists","up"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2352891695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":2352891695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":2352891695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2352891695}]},"public_addr":"172.21.15.154:6812/2352891695","cluster_addr":"172.21.15.154:6813/2352891695","heartbeat_back_addr":"172.21.15.154:6815/2352891695","heartbeat_front_addr":"172.21.15.154:6814/2352891695","state":["exists","up"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2304836889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":2304836889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":2304836889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2304836889}]},"public_addr":"172.21.15.204:6800/2304836889","cluster_addr":"172.21.15.204:6801/2304836889","heartbeat_back_addr":"172.21.15.204:6803/2304836889","heartbeat_front_addr":"172.21.15.204:6802/2304836889","state":["exists","up"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":58,"up_from":108,"up_thru":123,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":3237768415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3237768415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3237768415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":3237768415}]},"public_addr":"172.21.15.38:6809/3237768415","cluster_addr":"172.21.15.38:6810/3237768415","heartbeat_back_addr":"172.21.15.38:6812/3237768415","heartbeat_front_addr":"172.21.15.38:6811/3237768415","state":["exists","up"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":3727425991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":3727425991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":3727425991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":3727425991}]},"public_addr":"172.21.15.118:6812/3727425991","cluster_addr":"172.21.15.118:6813/3727425991","heartbeat_back_addr":"172.21.15.118:6815/3727425991","heartbeat_front_addr":"172.21.15.118:6814/3727425991","state":["exists","up"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":538351706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":538351706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":538351706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":538351706}]},"public_addr":"172.21.15.154:6804/538351706","cluster_addr":"172.21.15.154:6805/538351706","heartbeat_back_addr":"172.21.15.154:6807/538351706","heartbeat_front_addr":"172.21.15.154:6806/538351706","state":["exists","up"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":501019151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":501019151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":501019151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":501019151}]},"public_addr":"172.21.15.204:6812/501019151","cluster_addr":"172.21.15.204:6813/501019151","heartbeat_back_addr":"172.21.15.204:6815/501019151","heartbeat_front_addr":"172.21.15.204:6814/501019151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:22.298728+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:29.906048+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:24.237851+0000","dead_epoch":41},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:28.159438+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:27.889932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:21.167035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:28.767016+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:38.597935+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:26.703342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:37.877924+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:34.475326+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:33.370280+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:13.428589+0000","dead_epoch":59},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:38.011256+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:46.313159+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:09.456441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.7","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:35:03.637 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T11:35:03.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:35:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:03.938 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":155,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:35:03.295526+0000","last_up_change":"2024-09-13T11:34:25.125921+0000","last_in_change":"2024-09-13T11:34:31.396117+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:32:02.947125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:32:05.120118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T11:32:17.683275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":7.940000057220459,"score_stable":7.940000057220459,"optimal_score":0.62999999523162842,"raw_score_acting":5,"raw_score_stable":5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2677506544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2677506544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2677506544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":2677506544}]},"public_addr":"172.21.15.38:6805/2677506544","cluster_addr":"172.21.15.38:6806/2677506544","heartbeat_back_addr":"172.21.15.38:6808/2677506544","heartbeat_front_addr":"172.21.15.38:6807/2677506544","state":["exists","up"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":3561806747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":3561806747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":3561806747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":3561806747}]},"public_addr":"172.21.15.118:6808/3561806747","cluster_addr":"172.21.15.118:6809/3561806747","heartbeat_back_addr":"172.21.15.118:6811/3561806747","heartbeat_front_addr":"172.21.15.118:6810/3561806747","state":["exists","up"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":39,"up_from":116,"up_thru":0,"down_at":41,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3302524927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":3302524927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":3302524927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3302524927}]},"public_addr":"172.21.15.154:6800/3302524927","cluster_addr":"172.21.15.154:6801/3302524927","heartbeat_back_addr":"172.21.15.154:6803/3302524927","heartbeat_front_addr":"172.21.15.154:6802/3302524927","state":["exists","up"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2326505959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":2326505959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":2326505959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2326505959}]},"public_addr":"172.21.15.204:6804/2326505959","cluster_addr":"172.21.15.204:6805/2326505959","heartbeat_back_addr":"172.21.15.204:6807/2326505959","heartbeat_front_addr":"172.21.15.204:6806/2326505959","state":["exists","up"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":3107658120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3107658120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3107658120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":3107658120}]},"public_addr":"172.21.15.38:6801/3107658120","cluster_addr":"172.21.15.38:6802/3107658120","heartbeat_back_addr":"172.21.15.38:6804/3107658120","heartbeat_front_addr":"172.21.15.38:6803/3107658120","state":["exists","up"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3800782738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":3800782738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":3800782738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":3800782738}]},"public_addr":"172.21.15.118:6800/3800782738","cluster_addr":"172.21.15.118:6801/3800782738","heartbeat_back_addr":"172.21.15.118:6803/3800782738","heartbeat_front_addr":"172.21.15.118:6802/3800782738","state":["exists","up"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3986362716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3986362716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3986362716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3986362716}]},"public_addr":"172.21.15.154:6808/3986362716","cluster_addr":"172.21.15.154:6809/3986362716","heartbeat_back_addr":"172.21.15.154:6811/3986362716","heartbeat_front_addr":"172.21.15.154:6810/3986362716","state":["exists","up"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1978982202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":1978982202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":1978982202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1978982202}]},"public_addr":"172.21.15.204:6808/1978982202","cluster_addr":"172.21.15.204:6809/1978982202","heartbeat_back_addr":"172.21.15.204:6811/1978982202","heartbeat_front_addr":"172.21.15.204:6810/1978982202","state":["exists","up"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3608171775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3608171775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3608171775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3608171775}]},"public_addr":"172.21.15.38:6813/3608171775","cluster_addr":"172.21.15.38:6814/3608171775","heartbeat_back_addr":"172.21.15.38:6816/3608171775","heartbeat_front_addr":"172.21.15.38:6815/3608171775","state":["exists","up"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2580659844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2580659844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":2580659844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2580659844}]},"public_addr":"172.21.15.118:6804/2580659844","cluster_addr":"172.21.15.118:6805/2580659844","heartbeat_back_addr":"172.21.15.118:6807/2580659844","heartbeat_front_addr":"172.21.15.118:6806/2580659844","state":["exists","up"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2352891695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":2352891695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":2352891695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2352891695}]},"public_addr":"172.21.15.154:6812/2352891695","cluster_addr":"172.21.15.154:6813/2352891695","heartbeat_back_addr":"172.21.15.154:6815/2352891695","heartbeat_front_addr":"172.21.15.154:6814/2352891695","state":["exists","up"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2304836889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":2304836889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":2304836889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2304836889}]},"public_addr":"172.21.15.204:6800/2304836889","cluster_addr":"172.21.15.204:6801/2304836889","heartbeat_back_addr":"172.21.15.204:6803/2304836889","heartbeat_front_addr":"172.21.15.204:6802/2304836889","state":["exists","up"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":58,"up_from":108,"up_thru":123,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":3237768415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3237768415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3237768415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":3237768415}]},"public_addr":"172.21.15.38:6809/3237768415","cluster_addr":"172.21.15.38:6810/3237768415","heartbeat_back_addr":"172.21.15.38:6812/3237768415","heartbeat_front_addr":"172.21.15.38:6811/3237768415","state":["exists","up"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":3727425991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":3727425991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":3727425991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":3727425991}]},"public_addr":"172.21.15.118:6812/3727425991","cluster_addr":"172.21.15.118:6813/3727425991","heartbeat_back_addr":"172.21.15.118:6815/3727425991","heartbeat_front_addr":"172.21.15.118:6814/3727425991","state":["exists","up"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":538351706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":538351706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":538351706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":538351706}]},"public_addr":"172.21.15.154:6804/538351706","cluster_addr":"172.21.15.154:6805/538351706","heartbeat_back_addr":"172.21.15.154:6807/538351706","heartbeat_front_addr":"172.21.15.154:6806/538351706","state":["exists","up"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":501019151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":501019151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":501019151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":501019151}]},"public_addr":"172.21.15.204:6812/501019151","cluster_addr":"172.21.15.204:6813/501019151","heartbeat_back_addr":"172.21.15.204:6815/501019151","heartbeat_front_addr":"172.21.15.204:6814/501019151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:22.298728+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:29.906048+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:24.237851+0000","dead_epoch":41},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:28.159438+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:27.889932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:21.167035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:28.767016+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:38.597935+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:26.703342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:37.877924+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:34.475326+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:33.370280+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:13.428589+0000","dead_epoch":59},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:38.011256+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:46.313159+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:09.456441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.7","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:35:03.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T11:35:03.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T11:35:03.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T11:35:03.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T11:35:03.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T11:35:03.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T11:35:03.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T11:35:03.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T11:35:03.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-13T11:35:03.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-13T11:35:03.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-13T11:35:03.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-13T11:35:03.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-13T11:35:03.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-13T11:35:03.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-13T11:35:03.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-13T11:35:04.327 INFO:teuthology.orchestra.run.smithi038.stdout:498216206346 2024-09-13T11:35:04.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T11:35:04.426 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:04.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T11:35:04.443 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:04.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T11:35:04.445 INFO:teuthology.orchestra.run.smithi038.stdout:463856467980 2024-09-13T11:35:04.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-13T11:35:04.448 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:04.448 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:04.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T11:35:04.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T11:35:04.455 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:04.455 INFO:teuthology.orchestra.run.smithi038.stdout:98784247848 2024-09-13T11:35:04.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T11:35:04.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T11:35:04.469 INFO:teuthology.orchestra.run.smithi038.stdout:98784247848 2024-09-13T11:35:04.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-13T11:35:04.477 INFO:teuthology.orchestra.run.smithi038.stdout:90194313255 2024-09-13T11:35:04.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-13T11:35:04.496 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:04.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-13T11:35:04.499 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:04.501 INFO:teuthology.orchestra.run.smithi038.stdout:98784247849 2024-09-13T11:35:04.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-13T11:35:04.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-13T11:35:04.506 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:04.506 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:04.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-13T11:35:04.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T11:35:04.524 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:04.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-13T11:35:05.343 INFO:teuthology.orchestra.run.smithi038.stdout:498216206346 2024-09-13T11:35:05.371 INFO:tasks.ceph.ceph_manager.ceph:need seq 498216206346 got 498216206346 for osd.2 2024-09-13T11:35:05.372 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.424 INFO:teuthology.orchestra.run.smithi038.stdout:463856467980 2024-09-13T11:35:05.444 INFO:tasks.ceph.ceph_manager.ceph:need seq 463856467980 got 463856467980 for osd.12 2024-09-13T11:35:05.444 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.597 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:05.631 INFO:teuthology.orchestra.run.smithi038.stdout:98784247848 2024-09-13T11:35:05.638 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:05.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280552 got 94489280552 for osd.3 2024-09-13T11:35:05.639 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.655 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247848 got 98784247848 for osd.13 2024-09-13T11:35:05.655 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280552 got 94489280552 for osd.8 2024-09-13T11:35:05.669 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:05.669 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.691 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:05.716 INFO:teuthology.orchestra.run.smithi038.stdout:98784247848 2024-09-13T11:35:05.716 INFO:teuthology.orchestra.run.smithi038.stdout:90194313255 2024-09-13T11:35:05.717 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:05.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280551 got 94489280551 for osd.4 2024-09-13T11:35:05.718 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.720 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:05.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280551 got 94489280551 for osd.0 2024-09-13T11:35:05.721 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.741 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313255 got 90194313255 for osd.11 2024-09-13T11:35:05.741 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280551 got 94489280551 for osd.14 2024-09-13T11:35:05.742 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247848 got 98784247848 for osd.7 2024-09-13T11:35:05.743 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.749 INFO:teuthology.orchestra.run.smithi038.stdout:98784247849 2024-09-13T11:35:05.753 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280551 got 94489280551 for osd.9 2024-09-13T11:35:05.753 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.765 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:05.767 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247849 got 98784247849 for osd.15 2024-09-13T11:35:05.767 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:05.767 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.769 INFO:teuthology.orchestra.run.smithi038.stdout:94489280551 2024-09-13T11:35:05.778 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280552 got 94489280552 for osd.6 2024-09-13T11:35:05.778 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280551 got 94489280551 for osd.5 2024-09-13T11:35:05.785 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280551 got 94489280551 for osd.1 2024-09-13T11:35:05.786 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.786 INFO:teuthology.orchestra.run.smithi038.stdout:94489280552 2024-09-13T11:35:05.802 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280552 got 94489280552 for osd.10 2024-09-13T11:35:05.802 DEBUG:teuthology.parallel:result is None 2024-09-13T11:35:05.802 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-13T11:35:05.802 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:35:06.143 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:06.144 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:35:06.155 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-13T11:35:05.295792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112981749,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":1525,"num_read_kb":35747,"num_write":2106,"num_write_kb":392796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":228073177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":852584,"kb_used_data":351140,"kb_used_omap":36,"kb_used_meta":501147,"kb_avail":1509096856,"statfs":{"total":1546188226560,"available":1545315180544,"internally_reserved":0,"allocated":359567360,"data_stored":348872591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37303,"internal_metadata":513175113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.329443"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":310,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980941+0000","last_change":"2024-09-13T11:34:55.901795+0000","last_active":"2024-09-13T11:35:03.980941+0000","last_peered":"2024-09-13T11:35:03.980941+0000","last_clean":"2024-09-13T11:35:03.980941+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:35:03.980941+0000","last_undegraded":"2024-09-13T11:35:03.980941+0000","last_fullsized":"2024-09-13T11:35:03.980941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:55.901697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:55.901697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:25.291243+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.302059+0000","last_change":"2024-09-13T11:35:00.411948+0000","last_active":"2024-09-13T11:35:03.302059+0000","last_peered":"2024-09-13T11:35:03.302059+0000","last_clean":"2024-09-13T11:35:03.302059+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:35:03.302059+0000","last_undegraded":"2024-09-13T11:35:03.302059+0000","last_fullsized":"2024-09-13T11:35:03.302059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:00.411854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:35:00.411854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:19.314223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.303126+0000","last_change":"2024-09-13T11:34:26.615834+0000","last_active":"2024-09-13T11:35:03.303126+0000","last_peered":"2024-09-13T11:35:03.303126+0000","last_clean":"2024-09-13T11:35:03.303126+0000","last_became_active":"2024-09-13T11:34:26.221059+0000","last_became_peered":"2024-09-13T11:34:26.221059+0000","last_unstale":"2024-09-13T11:35:03.303126+0000","last_undegraded":"2024-09-13T11:35:03.303126+0000","last_fullsized":"2024-09-13T11:35:03.303126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:26.615729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:26.615729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:27.081593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980670+0000","last_change":"2024-09-13T11:34:18.999684+0000","last_active":"2024-09-13T11:35:03.980670+0000","last_peered":"2024-09-13T11:35:03.980670+0000","last_clean":"2024-09-13T11:35:03.980670+0000","last_became_active":"2024-09-13T11:34:15.290566+0000","last_became_peered":"2024-09-13T11:34:15.290566+0000","last_unstale":"2024-09-13T11:35:03.980670+0000","last_undegraded":"2024-09-13T11:35:03.980670+0000","last_fullsized":"2024-09-13T11:35:03.980670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:18.999603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:18.999603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:37.998553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"101'356","reported_seq":871,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299362+0000","last_change":"2024-09-13T11:34:37.797878+0000","last_active":"2024-09-13T11:35:03.299362+0000","last_peered":"2024-09-13T11:35:03.299362+0000","last_clean":"2024-09-13T11:35:03.299362+0000","last_became_active":"2024-09-13T11:34:28.142159+0000","last_became_peered":"2024-09-13T11:34:28.142159+0000","last_unstale":"2024-09-13T11:35:03.299362+0000","last_undegraded":"2024-09-13T11:35:03.299362+0000","last_fullsized":"2024-09-13T11:35:03.299362+0000","mapping_epoch":118,"log_start":"101'353","ondisk_log_start":"101'353","created":30,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'356","last_scrub_stamp":"2024-09-13T11:34:37.797738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:37.797738+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":362,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:55.713319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968389,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":353,"num_read_kb":9902,"num_write":469,"num_write_kb":89073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":40081204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2],"acting":[13,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":313,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981203+0000","last_change":"2024-09-13T11:34:38.032423+0000","last_active":"2024-09-13T11:35:03.981203+0000","last_peered":"2024-09-13T11:35:03.981203+0000","last_clean":"2024-09-13T11:35:03.981203+0000","last_became_active":"2024-09-13T11:34:30.146538+0000","last_became_peered":"2024-09-13T11:34:30.146538+0000","last_unstale":"2024-09-13T11:35:03.981203+0000","last_undegraded":"2024-09-13T11:35:03.981203+0000","last_fullsized":"2024-09-13T11:35:03.981203+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:34:38.032294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:38.032294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.425169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"101'346","reported_seq":887,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.799882+0000","last_change":"2024-09-13T11:34:32.417574+0000","last_active":"2024-09-13T11:35:03.799882+0000","last_peered":"2024-09-13T11:35:03.799882+0000","last_clean":"2024-09-13T11:35:03.799882+0000","last_became_active":"2024-09-13T11:34:32.417122+0000","last_became_peered":"2024-09-13T11:34:32.417122+0000","last_unstale":"2024-09-13T11:35:03.799882+0000","last_undegraded":"2024-09-13T11:35:03.799882+0000","last_fullsized":"2024-09-13T11:35:03.799882+0000","mapping_epoch":123,"log_start":"101'344","ondisk_log_start":"101'344","created":30,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'346","last_scrub_stamp":"2024-09-13T11:34:23.260748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:23.260748+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":350,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:40.949534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326473,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":312,"num_read_kb":13587,"num_write":452,"num_write_kb":94522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":46662709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299281+0000","last_change":"2024-09-13T11:34:42.738098+0000","last_active":"2024-09-13T11:35:03.299281+0000","last_peered":"2024-09-13T11:35:03.299281+0000","last_clean":"2024-09-13T11:35:03.299281+0000","last_became_active":"2024-09-13T11:34:30.143931+0000","last_became_peered":"2024-09-13T11:34:30.143931+0000","last_unstale":"2024-09-13T11:35:03.299281+0000","last_undegraded":"2024-09-13T11:35:03.299281+0000","last_fullsized":"2024-09-13T11:35:03.299281+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:42.738023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:42.738023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:51.864419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"101'387","reported_seq":1045,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.075065+0000","last_change":"2024-09-13T11:35:04.075065+0000","last_active":"2024-09-13T11:35:04.075065+0000","last_peered":"2024-09-13T11:35:04.075065+0000","last_clean":"2024-09-13T11:35:04.075065+0000","last_became_active":"2024-09-13T11:34:35.426908+0000","last_became_peered":"2024-09-13T11:34:35.426908+0000","last_unstale":"2024-09-13T11:35:04.075065+0000","last_undegraded":"2024-09-13T11:35:04.075065+0000","last_fullsized":"2024-09-13T11:35:04.075065+0000","mapping_epoch":126,"log_start":"101'384","ondisk_log_start":"101'384","created":30,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'387","last_scrub_stamp":"2024-09-13T11:35:04.074861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:35:04.074861+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":399,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:05.177977+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":374,"num_read_kb":6471,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":86512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981157+0000","last_change":"2024-09-13T11:34:51.125711+0000","last_active":"2024-09-13T11:35:03.981157+0000","last_peered":"2024-09-13T11:35:03.981157+0000","last_clean":"2024-09-13T11:35:03.981157+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:35:03.981157+0000","last_undegraded":"2024-09-13T11:35:03.981157+0000","last_fullsized":"2024-09-13T11:35:03.981157+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:51.125634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.125634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.787417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"101'422","reported_seq":938,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299314+0000","last_change":"2024-09-13T11:34:17.298237+0000","last_active":"2024-09-13T11:35:03.299314+0000","last_peered":"2024-09-13T11:35:03.299314+0000","last_clean":"2024-09-13T11:35:03.299314+0000","last_became_active":"2024-09-13T11:34:17.297909+0000","last_became_peered":"2024-09-13T11:34:17.297909+0000","last_unstale":"2024-09-13T11:35:03.299314+0000","last_undegraded":"2024-09-13T11:35:03.299314+0000","last_fullsized":"2024-09-13T11:35:03.299314+0000","mapping_epoch":110,"log_start":"101'418","ondisk_log_start":"101'418","created":30,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'422","last_scrub_stamp":"2024-09-13T11:34:11.938572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:11.938572+0000","objects_scrubbed":40,"log_size":4,"log_dups_size":433,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:32.609882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328092,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":400,"num_read_kb":5714,"num_write":543,"num_write_kb":103181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53636490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3],"acting":[13,12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":88,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.255512+0000","last_change":"2024-09-13T11:34:51.782015+0000","last_active":"2024-09-13T11:35:04.255512+0000","last_peered":"2024-09-13T11:35:04.255512+0000","last_clean":"2024-09-13T11:35:04.255512+0000","last_became_active":"2024-09-13T11:34:30.143993+0000","last_became_peered":"2024-09-13T11:34:30.143993+0000","last_unstale":"2024-09-13T11:35:04.255512+0000","last_undegraded":"2024-09-13T11:35:04.255512+0000","last_fullsized":"2024-09-13T11:35:04.255512+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:34:51.781912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.781912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:02.961286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":413,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299291+0000","last_change":"2024-09-13T11:34:46.802627+0000","last_active":"2024-09-13T11:35:03.299291+0000","last_peered":"2024-09-13T11:35:03.299291+0000","last_clean":"2024-09-13T11:35:03.299291+0000","last_became_active":"2024-09-13T11:34:01.289125+0000","last_became_peered":"2024-09-13T11:34:01.289125+0000","last_unstale":"2024-09-13T11:35:03.299291+0000","last_undegraded":"2024-09-13T11:35:03.299291+0000","last_fullsized":"2024-09-13T11:35:03.299291+0000","mapping_epoch":100,"log_start":"25'110","ondisk_log_start":"25'110","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:34:46.802521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:34:46.802521+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":110,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:03.688704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":112391362,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1439,"num_read_kb":35674,"num_write":1959,"num_write_kb":390122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":226892403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389038080,"data_stored":387751798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71552,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94300288,"statfs":{"total":96636764160,"available":96563494912,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84876,"kb_used_data":49532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286964,"statfs":{"total":96636764160,"available":96549851136,"internally_reserved":0,"allocated":50720768,"data_stored":49947521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58256,"kb_used_data":27072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313584,"statfs":{"total":96636764160,"available":96577110016,"internally_reserved":0,"allocated":27721728,"data_stored":27013733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1651040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67392,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304448,"statfs":{"total":96636764160,"available":96567754752,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":116,"seq":498216206346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52996,"kb_used_data":20272,"kb_used_omap":6,"kb_used_meta":32697,"kb_avail":94318844,"statfs":{"total":96636764160,"available":96582496256,"internally_reserved":0,"allocated":20758528,"data_stored":20062293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6811,"internal_metadata":33482085},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55248,"kb_used_data":24064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316592,"statfs":{"total":96636764160,"available":96580190208,"internally_reserved":0,"allocated":24641536,"data_stored":23979151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343204,"statfs":{"total":96636764160,"available":96607440896,"internally_reserved":0,"allocated":1650688,"data_stored":1053008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343212,"statfs":{"total":96636764160,"available":96607449088,"internally_reserved":0,"allocated":1642496,"data_stored":1045363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61000,"kb_used_data":29816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310840,"statfs":{"total":96636764160,"available":96574300160,"internally_reserved":0,"allocated":30531584,"data_stored":29829080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28648,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343192,"statfs":{"total":96636764160,"available":96607428608,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65148,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306692,"statfs":{"total":96636764160,"available":96570052608,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[2,3,4,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1654784,"data_stored":1053027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60988,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310852,"statfs":{"total":96636764160,"available":96574312448,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":108,"seq":463856467980,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86060,"kb_used_data":58076,"kb_used_omap":7,"kb_used_meta":27960,"kb_avail":94285780,"statfs":{"total":96636764160,"available":96548638720,"internally_reserved":0,"allocated":59469824,"data_stored":58715237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8160,"internal_metadata":28631072},"hb_peers":[1,2,3,5,6,7,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19107840,"data_stored":19009285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26083328,"data_stored":25968389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":57654565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":48463872,"data_stored":48296481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:35:06.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:35:06.454 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:35:06.467 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-13T11:35:05.295792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112981749,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":1525,"num_read_kb":35747,"num_write":2106,"num_write_kb":392796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":228073177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":852584,"kb_used_data":351140,"kb_used_omap":36,"kb_used_meta":501147,"kb_avail":1509096856,"statfs":{"total":1546188226560,"available":1545315180544,"internally_reserved":0,"allocated":359567360,"data_stored":348872591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37303,"internal_metadata":513175113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.329443"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":310,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980941+0000","last_change":"2024-09-13T11:34:55.901795+0000","last_active":"2024-09-13T11:35:03.980941+0000","last_peered":"2024-09-13T11:35:03.980941+0000","last_clean":"2024-09-13T11:35:03.980941+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:35:03.980941+0000","last_undegraded":"2024-09-13T11:35:03.980941+0000","last_fullsized":"2024-09-13T11:35:03.980941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:55.901697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:55.901697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:25.291243+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.302059+0000","last_change":"2024-09-13T11:35:00.411948+0000","last_active":"2024-09-13T11:35:03.302059+0000","last_peered":"2024-09-13T11:35:03.302059+0000","last_clean":"2024-09-13T11:35:03.302059+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:35:03.302059+0000","last_undegraded":"2024-09-13T11:35:03.302059+0000","last_fullsized":"2024-09-13T11:35:03.302059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:00.411854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:35:00.411854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:19.314223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.303126+0000","last_change":"2024-09-13T11:34:26.615834+0000","last_active":"2024-09-13T11:35:03.303126+0000","last_peered":"2024-09-13T11:35:03.303126+0000","last_clean":"2024-09-13T11:35:03.303126+0000","last_became_active":"2024-09-13T11:34:26.221059+0000","last_became_peered":"2024-09-13T11:34:26.221059+0000","last_unstale":"2024-09-13T11:35:03.303126+0000","last_undegraded":"2024-09-13T11:35:03.303126+0000","last_fullsized":"2024-09-13T11:35:03.303126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:26.615729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:26.615729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:27.081593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980670+0000","last_change":"2024-09-13T11:34:18.999684+0000","last_active":"2024-09-13T11:35:03.980670+0000","last_peered":"2024-09-13T11:35:03.980670+0000","last_clean":"2024-09-13T11:35:03.980670+0000","last_became_active":"2024-09-13T11:34:15.290566+0000","last_became_peered":"2024-09-13T11:34:15.290566+0000","last_unstale":"2024-09-13T11:35:03.980670+0000","last_undegraded":"2024-09-13T11:35:03.980670+0000","last_fullsized":"2024-09-13T11:35:03.980670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:18.999603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:18.999603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:37.998553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"101'356","reported_seq":871,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299362+0000","last_change":"2024-09-13T11:34:37.797878+0000","last_active":"2024-09-13T11:35:03.299362+0000","last_peered":"2024-09-13T11:35:03.299362+0000","last_clean":"2024-09-13T11:35:03.299362+0000","last_became_active":"2024-09-13T11:34:28.142159+0000","last_became_peered":"2024-09-13T11:34:28.142159+0000","last_unstale":"2024-09-13T11:35:03.299362+0000","last_undegraded":"2024-09-13T11:35:03.299362+0000","last_fullsized":"2024-09-13T11:35:03.299362+0000","mapping_epoch":118,"log_start":"101'353","ondisk_log_start":"101'353","created":30,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'356","last_scrub_stamp":"2024-09-13T11:34:37.797738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:37.797738+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":362,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:55.713319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968389,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":353,"num_read_kb":9902,"num_write":469,"num_write_kb":89073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":40081204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2],"acting":[13,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":313,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981203+0000","last_change":"2024-09-13T11:34:38.032423+0000","last_active":"2024-09-13T11:35:03.981203+0000","last_peered":"2024-09-13T11:35:03.981203+0000","last_clean":"2024-09-13T11:35:03.981203+0000","last_became_active":"2024-09-13T11:34:30.146538+0000","last_became_peered":"2024-09-13T11:34:30.146538+0000","last_unstale":"2024-09-13T11:35:03.981203+0000","last_undegraded":"2024-09-13T11:35:03.981203+0000","last_fullsized":"2024-09-13T11:35:03.981203+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:34:38.032294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:38.032294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.425169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"101'346","reported_seq":887,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.799882+0000","last_change":"2024-09-13T11:34:32.417574+0000","last_active":"2024-09-13T11:35:03.799882+0000","last_peered":"2024-09-13T11:35:03.799882+0000","last_clean":"2024-09-13T11:35:03.799882+0000","last_became_active":"2024-09-13T11:34:32.417122+0000","last_became_peered":"2024-09-13T11:34:32.417122+0000","last_unstale":"2024-09-13T11:35:03.799882+0000","last_undegraded":"2024-09-13T11:35:03.799882+0000","last_fullsized":"2024-09-13T11:35:03.799882+0000","mapping_epoch":123,"log_start":"101'344","ondisk_log_start":"101'344","created":30,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'346","last_scrub_stamp":"2024-09-13T11:34:23.260748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:23.260748+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":350,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:40.949534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326473,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":312,"num_read_kb":13587,"num_write":452,"num_write_kb":94522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":46662709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299281+0000","last_change":"2024-09-13T11:34:42.738098+0000","last_active":"2024-09-13T11:35:03.299281+0000","last_peered":"2024-09-13T11:35:03.299281+0000","last_clean":"2024-09-13T11:35:03.299281+0000","last_became_active":"2024-09-13T11:34:30.143931+0000","last_became_peered":"2024-09-13T11:34:30.143931+0000","last_unstale":"2024-09-13T11:35:03.299281+0000","last_undegraded":"2024-09-13T11:35:03.299281+0000","last_fullsized":"2024-09-13T11:35:03.299281+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:42.738023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:42.738023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:51.864419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"101'387","reported_seq":1045,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.075065+0000","last_change":"2024-09-13T11:35:04.075065+0000","last_active":"2024-09-13T11:35:04.075065+0000","last_peered":"2024-09-13T11:35:04.075065+0000","last_clean":"2024-09-13T11:35:04.075065+0000","last_became_active":"2024-09-13T11:34:35.426908+0000","last_became_peered":"2024-09-13T11:34:35.426908+0000","last_unstale":"2024-09-13T11:35:04.075065+0000","last_undegraded":"2024-09-13T11:35:04.075065+0000","last_fullsized":"2024-09-13T11:35:04.075065+0000","mapping_epoch":126,"log_start":"101'384","ondisk_log_start":"101'384","created":30,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'387","last_scrub_stamp":"2024-09-13T11:35:04.074861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:35:04.074861+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":399,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:05.177977+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":374,"num_read_kb":6471,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":86512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981157+0000","last_change":"2024-09-13T11:34:51.125711+0000","last_active":"2024-09-13T11:35:03.981157+0000","last_peered":"2024-09-13T11:35:03.981157+0000","last_clean":"2024-09-13T11:35:03.981157+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:35:03.981157+0000","last_undegraded":"2024-09-13T11:35:03.981157+0000","last_fullsized":"2024-09-13T11:35:03.981157+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:51.125634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.125634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.787417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"101'422","reported_seq":938,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299314+0000","last_change":"2024-09-13T11:34:17.298237+0000","last_active":"2024-09-13T11:35:03.299314+0000","last_peered":"2024-09-13T11:35:03.299314+0000","last_clean":"2024-09-13T11:35:03.299314+0000","last_became_active":"2024-09-13T11:34:17.297909+0000","last_became_peered":"2024-09-13T11:34:17.297909+0000","last_unstale":"2024-09-13T11:35:03.299314+0000","last_undegraded":"2024-09-13T11:35:03.299314+0000","last_fullsized":"2024-09-13T11:35:03.299314+0000","mapping_epoch":110,"log_start":"101'418","ondisk_log_start":"101'418","created":30,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'422","last_scrub_stamp":"2024-09-13T11:34:11.938572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:11.938572+0000","objects_scrubbed":40,"log_size":4,"log_dups_size":433,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:32.609882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328092,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":400,"num_read_kb":5714,"num_write":543,"num_write_kb":103181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53636490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3],"acting":[13,12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":88,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.255512+0000","last_change":"2024-09-13T11:34:51.782015+0000","last_active":"2024-09-13T11:35:04.255512+0000","last_peered":"2024-09-13T11:35:04.255512+0000","last_clean":"2024-09-13T11:35:04.255512+0000","last_became_active":"2024-09-13T11:34:30.143993+0000","last_became_peered":"2024-09-13T11:34:30.143993+0000","last_unstale":"2024-09-13T11:35:04.255512+0000","last_undegraded":"2024-09-13T11:35:04.255512+0000","last_fullsized":"2024-09-13T11:35:04.255512+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:34:51.781912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.781912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:02.961286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":413,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299291+0000","last_change":"2024-09-13T11:34:46.802627+0000","last_active":"2024-09-13T11:35:03.299291+0000","last_peered":"2024-09-13T11:35:03.299291+0000","last_clean":"2024-09-13T11:35:03.299291+0000","last_became_active":"2024-09-13T11:34:01.289125+0000","last_became_peered":"2024-09-13T11:34:01.289125+0000","last_unstale":"2024-09-13T11:35:03.299291+0000","last_undegraded":"2024-09-13T11:35:03.299291+0000","last_fullsized":"2024-09-13T11:35:03.299291+0000","mapping_epoch":100,"log_start":"25'110","ondisk_log_start":"25'110","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:34:46.802521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:34:46.802521+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":110,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:03.688704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":112391362,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1439,"num_read_kb":35674,"num_write":1959,"num_write_kb":390122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":226892403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389038080,"data_stored":387751798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71552,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94300288,"statfs":{"total":96636764160,"available":96563494912,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84876,"kb_used_data":49532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286964,"statfs":{"total":96636764160,"available":96549851136,"internally_reserved":0,"allocated":50720768,"data_stored":49947521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58256,"kb_used_data":27072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313584,"statfs":{"total":96636764160,"available":96577110016,"internally_reserved":0,"allocated":27721728,"data_stored":27013733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1651040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67392,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304448,"statfs":{"total":96636764160,"available":96567754752,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":116,"seq":498216206346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52996,"kb_used_data":20272,"kb_used_omap":6,"kb_used_meta":32697,"kb_avail":94318844,"statfs":{"total":96636764160,"available":96582496256,"internally_reserved":0,"allocated":20758528,"data_stored":20062293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6811,"internal_metadata":33482085},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55248,"kb_used_data":24064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316592,"statfs":{"total":96636764160,"available":96580190208,"internally_reserved":0,"allocated":24641536,"data_stored":23979151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343204,"statfs":{"total":96636764160,"available":96607440896,"internally_reserved":0,"allocated":1650688,"data_stored":1053008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343212,"statfs":{"total":96636764160,"available":96607449088,"internally_reserved":0,"allocated":1642496,"data_stored":1045363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61000,"kb_used_data":29816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310840,"statfs":{"total":96636764160,"available":96574300160,"internally_reserved":0,"allocated":30531584,"data_stored":29829080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28648,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343192,"statfs":{"total":96636764160,"available":96607428608,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65148,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306692,"statfs":{"total":96636764160,"available":96570052608,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[2,3,4,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1654784,"data_stored":1053027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60988,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310852,"statfs":{"total":96636764160,"available":96574312448,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":108,"seq":463856467980,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86060,"kb_used_data":58076,"kb_used_omap":7,"kb_used_meta":27960,"kb_avail":94285780,"statfs":{"total":96636764160,"available":96548638720,"internally_reserved":0,"allocated":59469824,"data_stored":58715237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8160,"internal_metadata":28631072},"hb_peers":[1,2,3,5,6,7,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19107840,"data_stored":19009285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26083328,"data_stored":25968389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":57654565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":48463872,"data_stored":48296481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:35:06.468 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-13T11:35:06.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-13T11:35:06.788 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T11:35:06.797 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T11:35:06.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:35:07.098 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:07.098 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:35:07.110 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-13T11:35:06.967909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112981749,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":1525,"num_read_kb":35747,"num_write":2106,"num_write_kb":392796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":228073177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":852584,"kb_used_data":351140,"kb_used_omap":36,"kb_used_meta":501147,"kb_avail":1509096856,"statfs":{"total":1546188226560,"available":1545315180544,"internally_reserved":0,"allocated":359567360,"data_stored":348872591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37303,"internal_metadata":513175113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.673077"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":310,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980941+0000","last_change":"2024-09-13T11:34:55.901795+0000","last_active":"2024-09-13T11:35:03.980941+0000","last_peered":"2024-09-13T11:35:03.980941+0000","last_clean":"2024-09-13T11:35:03.980941+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:35:03.980941+0000","last_undegraded":"2024-09-13T11:35:03.980941+0000","last_fullsized":"2024-09-13T11:35:03.980941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:55.901697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:55.901697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:25.291243+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.302059+0000","last_change":"2024-09-13T11:35:00.411948+0000","last_active":"2024-09-13T11:35:03.302059+0000","last_peered":"2024-09-13T11:35:03.302059+0000","last_clean":"2024-09-13T11:35:03.302059+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:35:03.302059+0000","last_undegraded":"2024-09-13T11:35:03.302059+0000","last_fullsized":"2024-09-13T11:35:03.302059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:00.411854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:35:00.411854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:19.314223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.303126+0000","last_change":"2024-09-13T11:34:26.615834+0000","last_active":"2024-09-13T11:35:03.303126+0000","last_peered":"2024-09-13T11:35:03.303126+0000","last_clean":"2024-09-13T11:35:03.303126+0000","last_became_active":"2024-09-13T11:34:26.221059+0000","last_became_peered":"2024-09-13T11:34:26.221059+0000","last_unstale":"2024-09-13T11:35:03.303126+0000","last_undegraded":"2024-09-13T11:35:03.303126+0000","last_fullsized":"2024-09-13T11:35:03.303126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:26.615729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:26.615729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:27.081593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980670+0000","last_change":"2024-09-13T11:34:18.999684+0000","last_active":"2024-09-13T11:35:03.980670+0000","last_peered":"2024-09-13T11:35:03.980670+0000","last_clean":"2024-09-13T11:35:03.980670+0000","last_became_active":"2024-09-13T11:34:15.290566+0000","last_became_peered":"2024-09-13T11:34:15.290566+0000","last_unstale":"2024-09-13T11:35:03.980670+0000","last_undegraded":"2024-09-13T11:35:03.980670+0000","last_fullsized":"2024-09-13T11:35:03.980670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:18.999603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:18.999603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:37.998553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"101'356","reported_seq":871,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299362+0000","last_change":"2024-09-13T11:34:37.797878+0000","last_active":"2024-09-13T11:35:03.299362+0000","last_peered":"2024-09-13T11:35:03.299362+0000","last_clean":"2024-09-13T11:35:03.299362+0000","last_became_active":"2024-09-13T11:34:28.142159+0000","last_became_peered":"2024-09-13T11:34:28.142159+0000","last_unstale":"2024-09-13T11:35:03.299362+0000","last_undegraded":"2024-09-13T11:35:03.299362+0000","last_fullsized":"2024-09-13T11:35:03.299362+0000","mapping_epoch":118,"log_start":"101'353","ondisk_log_start":"101'353","created":30,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'356","last_scrub_stamp":"2024-09-13T11:34:37.797738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:37.797738+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":362,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:55.713319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968389,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":353,"num_read_kb":9902,"num_write":469,"num_write_kb":89073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":40081204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2],"acting":[13,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":313,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981203+0000","last_change":"2024-09-13T11:34:38.032423+0000","last_active":"2024-09-13T11:35:03.981203+0000","last_peered":"2024-09-13T11:35:03.981203+0000","last_clean":"2024-09-13T11:35:03.981203+0000","last_became_active":"2024-09-13T11:34:30.146538+0000","last_became_peered":"2024-09-13T11:34:30.146538+0000","last_unstale":"2024-09-13T11:35:03.981203+0000","last_undegraded":"2024-09-13T11:35:03.981203+0000","last_fullsized":"2024-09-13T11:35:03.981203+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:34:38.032294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:38.032294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.425169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"101'346","reported_seq":887,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.799882+0000","last_change":"2024-09-13T11:34:32.417574+0000","last_active":"2024-09-13T11:35:03.799882+0000","last_peered":"2024-09-13T11:35:03.799882+0000","last_clean":"2024-09-13T11:35:03.799882+0000","last_became_active":"2024-09-13T11:34:32.417122+0000","last_became_peered":"2024-09-13T11:34:32.417122+0000","last_unstale":"2024-09-13T11:35:03.799882+0000","last_undegraded":"2024-09-13T11:35:03.799882+0000","last_fullsized":"2024-09-13T11:35:03.799882+0000","mapping_epoch":123,"log_start":"101'344","ondisk_log_start":"101'344","created":30,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'346","last_scrub_stamp":"2024-09-13T11:34:23.260748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:23.260748+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":350,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:40.949534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326473,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":312,"num_read_kb":13587,"num_write":452,"num_write_kb":94522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":46662709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299281+0000","last_change":"2024-09-13T11:34:42.738098+0000","last_active":"2024-09-13T11:35:03.299281+0000","last_peered":"2024-09-13T11:35:03.299281+0000","last_clean":"2024-09-13T11:35:03.299281+0000","last_became_active":"2024-09-13T11:34:30.143931+0000","last_became_peered":"2024-09-13T11:34:30.143931+0000","last_unstale":"2024-09-13T11:35:03.299281+0000","last_undegraded":"2024-09-13T11:35:03.299281+0000","last_fullsized":"2024-09-13T11:35:03.299281+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:42.738023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:42.738023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:51.864419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"101'387","reported_seq":1045,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.075065+0000","last_change":"2024-09-13T11:35:04.075065+0000","last_active":"2024-09-13T11:35:04.075065+0000","last_peered":"2024-09-13T11:35:04.075065+0000","last_clean":"2024-09-13T11:35:04.075065+0000","last_became_active":"2024-09-13T11:34:35.426908+0000","last_became_peered":"2024-09-13T11:34:35.426908+0000","last_unstale":"2024-09-13T11:35:04.075065+0000","last_undegraded":"2024-09-13T11:35:04.075065+0000","last_fullsized":"2024-09-13T11:35:04.075065+0000","mapping_epoch":126,"log_start":"101'384","ondisk_log_start":"101'384","created":30,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'387","last_scrub_stamp":"2024-09-13T11:35:04.074861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:35:04.074861+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":399,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:05.177977+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":374,"num_read_kb":6471,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":86512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981157+0000","last_change":"2024-09-13T11:34:51.125711+0000","last_active":"2024-09-13T11:35:03.981157+0000","last_peered":"2024-09-13T11:35:03.981157+0000","last_clean":"2024-09-13T11:35:03.981157+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:35:03.981157+0000","last_undegraded":"2024-09-13T11:35:03.981157+0000","last_fullsized":"2024-09-13T11:35:03.981157+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:51.125634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.125634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.787417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"101'422","reported_seq":938,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299314+0000","last_change":"2024-09-13T11:34:17.298237+0000","last_active":"2024-09-13T11:35:03.299314+0000","last_peered":"2024-09-13T11:35:03.299314+0000","last_clean":"2024-09-13T11:35:03.299314+0000","last_became_active":"2024-09-13T11:34:17.297909+0000","last_became_peered":"2024-09-13T11:34:17.297909+0000","last_unstale":"2024-09-13T11:35:03.299314+0000","last_undegraded":"2024-09-13T11:35:03.299314+0000","last_fullsized":"2024-09-13T11:35:03.299314+0000","mapping_epoch":110,"log_start":"101'418","ondisk_log_start":"101'418","created":30,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'422","last_scrub_stamp":"2024-09-13T11:34:11.938572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:11.938572+0000","objects_scrubbed":40,"log_size":4,"log_dups_size":433,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:32.609882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328092,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":400,"num_read_kb":5714,"num_write":543,"num_write_kb":103181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53636490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3],"acting":[13,12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":88,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.255512+0000","last_change":"2024-09-13T11:34:51.782015+0000","last_active":"2024-09-13T11:35:04.255512+0000","last_peered":"2024-09-13T11:35:04.255512+0000","last_clean":"2024-09-13T11:35:04.255512+0000","last_became_active":"2024-09-13T11:34:30.143993+0000","last_became_peered":"2024-09-13T11:34:30.143993+0000","last_unstale":"2024-09-13T11:35:04.255512+0000","last_undegraded":"2024-09-13T11:35:04.255512+0000","last_fullsized":"2024-09-13T11:35:04.255512+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:34:51.781912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.781912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:02.961286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":413,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299291+0000","last_change":"2024-09-13T11:34:46.802627+0000","last_active":"2024-09-13T11:35:03.299291+0000","last_peered":"2024-09-13T11:35:03.299291+0000","last_clean":"2024-09-13T11:35:03.299291+0000","last_became_active":"2024-09-13T11:34:01.289125+0000","last_became_peered":"2024-09-13T11:34:01.289125+0000","last_unstale":"2024-09-13T11:35:03.299291+0000","last_undegraded":"2024-09-13T11:35:03.299291+0000","last_fullsized":"2024-09-13T11:35:03.299291+0000","mapping_epoch":100,"log_start":"25'110","ondisk_log_start":"25'110","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:34:46.802521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:34:46.802521+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":110,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:03.688704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":112391362,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1439,"num_read_kb":35674,"num_write":1959,"num_write_kb":390122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":226892403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389038080,"data_stored":387751798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71552,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94300288,"statfs":{"total":96636764160,"available":96563494912,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84876,"kb_used_data":49532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286964,"statfs":{"total":96636764160,"available":96549851136,"internally_reserved":0,"allocated":50720768,"data_stored":49947521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58256,"kb_used_data":27072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313584,"statfs":{"total":96636764160,"available":96577110016,"internally_reserved":0,"allocated":27721728,"data_stored":27013733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1651040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67392,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304448,"statfs":{"total":96636764160,"available":96567754752,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":116,"seq":498216206346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52996,"kb_used_data":20272,"kb_used_omap":6,"kb_used_meta":32697,"kb_avail":94318844,"statfs":{"total":96636764160,"available":96582496256,"internally_reserved":0,"allocated":20758528,"data_stored":20062293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6811,"internal_metadata":33482085},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55248,"kb_used_data":24064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316592,"statfs":{"total":96636764160,"available":96580190208,"internally_reserved":0,"allocated":24641536,"data_stored":23979151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343204,"statfs":{"total":96636764160,"available":96607440896,"internally_reserved":0,"allocated":1650688,"data_stored":1053008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343212,"statfs":{"total":96636764160,"available":96607449088,"internally_reserved":0,"allocated":1642496,"data_stored":1045363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61000,"kb_used_data":29816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310840,"statfs":{"total":96636764160,"available":96574300160,"internally_reserved":0,"allocated":30531584,"data_stored":29829080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28648,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343192,"statfs":{"total":96636764160,"available":96607428608,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65148,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306692,"statfs":{"total":96636764160,"available":96570052608,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[2,3,4,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1654784,"data_stored":1053027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60988,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310852,"statfs":{"total":96636764160,"available":96574312448,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":108,"seq":463856467980,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86060,"kb_used_data":58076,"kb_used_omap":7,"kb_used_meta":27960,"kb_avail":94285780,"statfs":{"total":96636764160,"available":96548638720,"internally_reserved":0,"allocated":59469824,"data_stored":58715237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8160,"internal_metadata":28631072},"hb_peers":[1,2,3,5,6,7,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19107840,"data_stored":19009285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26083328,"data_stored":25968389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":57654565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":48463872,"data_stored":48296481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:35:07.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:35:07.410 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:35:07.423 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-13T11:35:06.967909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112981749,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":1525,"num_read_kb":35747,"num_write":2106,"num_write_kb":392796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":228073177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":852584,"kb_used_data":351140,"kb_used_omap":36,"kb_used_meta":501147,"kb_avail":1509096856,"statfs":{"total":1546188226560,"available":1545315180544,"internally_reserved":0,"allocated":359567360,"data_stored":348872591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37303,"internal_metadata":513175113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.673077"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":310,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980941+0000","last_change":"2024-09-13T11:34:55.901795+0000","last_active":"2024-09-13T11:35:03.980941+0000","last_peered":"2024-09-13T11:35:03.980941+0000","last_clean":"2024-09-13T11:35:03.980941+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:35:03.980941+0000","last_undegraded":"2024-09-13T11:35:03.980941+0000","last_fullsized":"2024-09-13T11:35:03.980941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:55.901697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:55.901697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:25.291243+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.302059+0000","last_change":"2024-09-13T11:35:00.411948+0000","last_active":"2024-09-13T11:35:03.302059+0000","last_peered":"2024-09-13T11:35:03.302059+0000","last_clean":"2024-09-13T11:35:03.302059+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:35:03.302059+0000","last_undegraded":"2024-09-13T11:35:03.302059+0000","last_fullsized":"2024-09-13T11:35:03.302059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:00.411854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:35:00.411854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:19.314223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.303126+0000","last_change":"2024-09-13T11:34:26.615834+0000","last_active":"2024-09-13T11:35:03.303126+0000","last_peered":"2024-09-13T11:35:03.303126+0000","last_clean":"2024-09-13T11:35:03.303126+0000","last_became_active":"2024-09-13T11:34:26.221059+0000","last_became_peered":"2024-09-13T11:34:26.221059+0000","last_unstale":"2024-09-13T11:35:03.303126+0000","last_undegraded":"2024-09-13T11:35:03.303126+0000","last_fullsized":"2024-09-13T11:35:03.303126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:26.615729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:26.615729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:27.081593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980670+0000","last_change":"2024-09-13T11:34:18.999684+0000","last_active":"2024-09-13T11:35:03.980670+0000","last_peered":"2024-09-13T11:35:03.980670+0000","last_clean":"2024-09-13T11:35:03.980670+0000","last_became_active":"2024-09-13T11:34:15.290566+0000","last_became_peered":"2024-09-13T11:34:15.290566+0000","last_unstale":"2024-09-13T11:35:03.980670+0000","last_undegraded":"2024-09-13T11:35:03.980670+0000","last_fullsized":"2024-09-13T11:35:03.980670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:18.999603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:18.999603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:37.998553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"101'356","reported_seq":871,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299362+0000","last_change":"2024-09-13T11:34:37.797878+0000","last_active":"2024-09-13T11:35:03.299362+0000","last_peered":"2024-09-13T11:35:03.299362+0000","last_clean":"2024-09-13T11:35:03.299362+0000","last_became_active":"2024-09-13T11:34:28.142159+0000","last_became_peered":"2024-09-13T11:34:28.142159+0000","last_unstale":"2024-09-13T11:35:03.299362+0000","last_undegraded":"2024-09-13T11:35:03.299362+0000","last_fullsized":"2024-09-13T11:35:03.299362+0000","mapping_epoch":118,"log_start":"101'353","ondisk_log_start":"101'353","created":30,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'356","last_scrub_stamp":"2024-09-13T11:34:37.797738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:37.797738+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":362,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:55.713319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968389,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":353,"num_read_kb":9902,"num_write":469,"num_write_kb":89073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":40081204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2],"acting":[13,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":313,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981203+0000","last_change":"2024-09-13T11:34:38.032423+0000","last_active":"2024-09-13T11:35:03.981203+0000","last_peered":"2024-09-13T11:35:03.981203+0000","last_clean":"2024-09-13T11:35:03.981203+0000","last_became_active":"2024-09-13T11:34:30.146538+0000","last_became_peered":"2024-09-13T11:34:30.146538+0000","last_unstale":"2024-09-13T11:35:03.981203+0000","last_undegraded":"2024-09-13T11:35:03.981203+0000","last_fullsized":"2024-09-13T11:35:03.981203+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:34:38.032294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:38.032294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.425169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"101'346","reported_seq":887,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.799882+0000","last_change":"2024-09-13T11:34:32.417574+0000","last_active":"2024-09-13T11:35:03.799882+0000","last_peered":"2024-09-13T11:35:03.799882+0000","last_clean":"2024-09-13T11:35:03.799882+0000","last_became_active":"2024-09-13T11:34:32.417122+0000","last_became_peered":"2024-09-13T11:34:32.417122+0000","last_unstale":"2024-09-13T11:35:03.799882+0000","last_undegraded":"2024-09-13T11:35:03.799882+0000","last_fullsized":"2024-09-13T11:35:03.799882+0000","mapping_epoch":123,"log_start":"101'344","ondisk_log_start":"101'344","created":30,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'346","last_scrub_stamp":"2024-09-13T11:34:23.260748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:23.260748+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":350,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:40.949534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326473,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":312,"num_read_kb":13587,"num_write":452,"num_write_kb":94522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":46662709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299281+0000","last_change":"2024-09-13T11:34:42.738098+0000","last_active":"2024-09-13T11:35:03.299281+0000","last_peered":"2024-09-13T11:35:03.299281+0000","last_clean":"2024-09-13T11:35:03.299281+0000","last_became_active":"2024-09-13T11:34:30.143931+0000","last_became_peered":"2024-09-13T11:34:30.143931+0000","last_unstale":"2024-09-13T11:35:03.299281+0000","last_undegraded":"2024-09-13T11:35:03.299281+0000","last_fullsized":"2024-09-13T11:35:03.299281+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:42.738023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:42.738023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:51.864419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"101'387","reported_seq":1045,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.075065+0000","last_change":"2024-09-13T11:35:04.075065+0000","last_active":"2024-09-13T11:35:04.075065+0000","last_peered":"2024-09-13T11:35:04.075065+0000","last_clean":"2024-09-13T11:35:04.075065+0000","last_became_active":"2024-09-13T11:34:35.426908+0000","last_became_peered":"2024-09-13T11:34:35.426908+0000","last_unstale":"2024-09-13T11:35:04.075065+0000","last_undegraded":"2024-09-13T11:35:04.075065+0000","last_fullsized":"2024-09-13T11:35:04.075065+0000","mapping_epoch":126,"log_start":"101'384","ondisk_log_start":"101'384","created":30,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'387","last_scrub_stamp":"2024-09-13T11:35:04.074861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:35:04.074861+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":399,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:05.177977+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":374,"num_read_kb":6471,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":86512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981157+0000","last_change":"2024-09-13T11:34:51.125711+0000","last_active":"2024-09-13T11:35:03.981157+0000","last_peered":"2024-09-13T11:35:03.981157+0000","last_clean":"2024-09-13T11:35:03.981157+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:35:03.981157+0000","last_undegraded":"2024-09-13T11:35:03.981157+0000","last_fullsized":"2024-09-13T11:35:03.981157+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:51.125634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.125634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.787417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"101'422","reported_seq":938,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299314+0000","last_change":"2024-09-13T11:34:17.298237+0000","last_active":"2024-09-13T11:35:03.299314+0000","last_peered":"2024-09-13T11:35:03.299314+0000","last_clean":"2024-09-13T11:35:03.299314+0000","last_became_active":"2024-09-13T11:34:17.297909+0000","last_became_peered":"2024-09-13T11:34:17.297909+0000","last_unstale":"2024-09-13T11:35:03.299314+0000","last_undegraded":"2024-09-13T11:35:03.299314+0000","last_fullsized":"2024-09-13T11:35:03.299314+0000","mapping_epoch":110,"log_start":"101'418","ondisk_log_start":"101'418","created":30,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'422","last_scrub_stamp":"2024-09-13T11:34:11.938572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:11.938572+0000","objects_scrubbed":40,"log_size":4,"log_dups_size":433,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:32.609882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328092,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":400,"num_read_kb":5714,"num_write":543,"num_write_kb":103181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53636490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3],"acting":[13,12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":88,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.255512+0000","last_change":"2024-09-13T11:34:51.782015+0000","last_active":"2024-09-13T11:35:04.255512+0000","last_peered":"2024-09-13T11:35:04.255512+0000","last_clean":"2024-09-13T11:35:04.255512+0000","last_became_active":"2024-09-13T11:34:30.143993+0000","last_became_peered":"2024-09-13T11:34:30.143993+0000","last_unstale":"2024-09-13T11:35:04.255512+0000","last_undegraded":"2024-09-13T11:35:04.255512+0000","last_fullsized":"2024-09-13T11:35:04.255512+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:34:51.781912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.781912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:02.961286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":413,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299291+0000","last_change":"2024-09-13T11:34:46.802627+0000","last_active":"2024-09-13T11:35:03.299291+0000","last_peered":"2024-09-13T11:35:03.299291+0000","last_clean":"2024-09-13T11:35:03.299291+0000","last_became_active":"2024-09-13T11:34:01.289125+0000","last_became_peered":"2024-09-13T11:34:01.289125+0000","last_unstale":"2024-09-13T11:35:03.299291+0000","last_undegraded":"2024-09-13T11:35:03.299291+0000","last_fullsized":"2024-09-13T11:35:03.299291+0000","mapping_epoch":100,"log_start":"25'110","ondisk_log_start":"25'110","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:34:46.802521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:34:46.802521+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":110,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:03.688704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":112391362,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1439,"num_read_kb":35674,"num_write":1959,"num_write_kb":390122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":226892403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389038080,"data_stored":387751798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71552,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94300288,"statfs":{"total":96636764160,"available":96563494912,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84876,"kb_used_data":49532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286964,"statfs":{"total":96636764160,"available":96549851136,"internally_reserved":0,"allocated":50720768,"data_stored":49947521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58256,"kb_used_data":27072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313584,"statfs":{"total":96636764160,"available":96577110016,"internally_reserved":0,"allocated":27721728,"data_stored":27013733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1651040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67392,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304448,"statfs":{"total":96636764160,"available":96567754752,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":116,"seq":498216206346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52996,"kb_used_data":20272,"kb_used_omap":6,"kb_used_meta":32697,"kb_avail":94318844,"statfs":{"total":96636764160,"available":96582496256,"internally_reserved":0,"allocated":20758528,"data_stored":20062293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6811,"internal_metadata":33482085},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55248,"kb_used_data":24064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316592,"statfs":{"total":96636764160,"available":96580190208,"internally_reserved":0,"allocated":24641536,"data_stored":23979151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343204,"statfs":{"total":96636764160,"available":96607440896,"internally_reserved":0,"allocated":1650688,"data_stored":1053008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343212,"statfs":{"total":96636764160,"available":96607449088,"internally_reserved":0,"allocated":1642496,"data_stored":1045363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61000,"kb_used_data":29816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310840,"statfs":{"total":96636764160,"available":96574300160,"internally_reserved":0,"allocated":30531584,"data_stored":29829080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28648,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343192,"statfs":{"total":96636764160,"available":96607428608,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65148,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306692,"statfs":{"total":96636764160,"available":96570052608,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[2,3,4,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1654784,"data_stored":1053027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60988,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310852,"statfs":{"total":96636764160,"available":96574312448,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":108,"seq":463856467980,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86060,"kb_used_data":58076,"kb_used_omap":7,"kb_used_meta":27960,"kb_avail":94285780,"statfs":{"total":96636764160,"available":96548638720,"internally_reserved":0,"allocated":59469824,"data_stored":58715237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8160,"internal_metadata":28631072},"hb_peers":[1,2,3,5,6,7,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19107840,"data_stored":19009285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26083328,"data_stored":25968389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":57654565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":48463872,"data_stored":48296481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:35:07.424 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T11:35:07.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:35:07.723 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:07.724 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:35:07.736 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-13T11:35:06.967909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112981749,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":1525,"num_read_kb":35747,"num_write":2106,"num_write_kb":392796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":228073177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":852584,"kb_used_data":351140,"kb_used_omap":36,"kb_used_meta":501147,"kb_avail":1509096856,"statfs":{"total":1546188226560,"available":1545315180544,"internally_reserved":0,"allocated":359567360,"data_stored":348872591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37303,"internal_metadata":513175113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.673077"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":310,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980941+0000","last_change":"2024-09-13T11:34:55.901795+0000","last_active":"2024-09-13T11:35:03.980941+0000","last_peered":"2024-09-13T11:35:03.980941+0000","last_clean":"2024-09-13T11:35:03.980941+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:35:03.980941+0000","last_undegraded":"2024-09-13T11:35:03.980941+0000","last_fullsized":"2024-09-13T11:35:03.980941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:55.901697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:55.901697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:25.291243+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.302059+0000","last_change":"2024-09-13T11:35:00.411948+0000","last_active":"2024-09-13T11:35:03.302059+0000","last_peered":"2024-09-13T11:35:03.302059+0000","last_clean":"2024-09-13T11:35:03.302059+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:35:03.302059+0000","last_undegraded":"2024-09-13T11:35:03.302059+0000","last_fullsized":"2024-09-13T11:35:03.302059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:00.411854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:35:00.411854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:19.314223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.303126+0000","last_change":"2024-09-13T11:34:26.615834+0000","last_active":"2024-09-13T11:35:03.303126+0000","last_peered":"2024-09-13T11:35:03.303126+0000","last_clean":"2024-09-13T11:35:03.303126+0000","last_became_active":"2024-09-13T11:34:26.221059+0000","last_became_peered":"2024-09-13T11:34:26.221059+0000","last_unstale":"2024-09-13T11:35:03.303126+0000","last_undegraded":"2024-09-13T11:35:03.303126+0000","last_fullsized":"2024-09-13T11:35:03.303126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:26.615729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:26.615729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:27.081593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":292,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.980670+0000","last_change":"2024-09-13T11:34:18.999684+0000","last_active":"2024-09-13T11:35:03.980670+0000","last_peered":"2024-09-13T11:35:03.980670+0000","last_clean":"2024-09-13T11:35:03.980670+0000","last_became_active":"2024-09-13T11:34:15.290566+0000","last_became_peered":"2024-09-13T11:34:15.290566+0000","last_unstale":"2024-09-13T11:35:03.980670+0000","last_undegraded":"2024-09-13T11:35:03.980670+0000","last_fullsized":"2024-09-13T11:35:03.980670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:18.999603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:18.999603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:37.998553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"101'356","reported_seq":871,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299362+0000","last_change":"2024-09-13T11:34:37.797878+0000","last_active":"2024-09-13T11:35:03.299362+0000","last_peered":"2024-09-13T11:35:03.299362+0000","last_clean":"2024-09-13T11:35:03.299362+0000","last_became_active":"2024-09-13T11:34:28.142159+0000","last_became_peered":"2024-09-13T11:34:28.142159+0000","last_unstale":"2024-09-13T11:35:03.299362+0000","last_undegraded":"2024-09-13T11:35:03.299362+0000","last_fullsized":"2024-09-13T11:35:03.299362+0000","mapping_epoch":118,"log_start":"101'353","ondisk_log_start":"101'353","created":30,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'356","last_scrub_stamp":"2024-09-13T11:34:37.797738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:37.797738+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":362,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:55.713319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968389,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":353,"num_read_kb":9902,"num_write":469,"num_write_kb":89073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":40081204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2],"acting":[13,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":313,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981203+0000","last_change":"2024-09-13T11:34:38.032423+0000","last_active":"2024-09-13T11:35:03.981203+0000","last_peered":"2024-09-13T11:35:03.981203+0000","last_clean":"2024-09-13T11:35:03.981203+0000","last_became_active":"2024-09-13T11:34:30.146538+0000","last_became_peered":"2024-09-13T11:34:30.146538+0000","last_unstale":"2024-09-13T11:35:03.981203+0000","last_undegraded":"2024-09-13T11:35:03.981203+0000","last_fullsized":"2024-09-13T11:35:03.981203+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:34:38.032294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:38.032294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.425169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"101'346","reported_seq":887,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.799882+0000","last_change":"2024-09-13T11:34:32.417574+0000","last_active":"2024-09-13T11:35:03.799882+0000","last_peered":"2024-09-13T11:35:03.799882+0000","last_clean":"2024-09-13T11:35:03.799882+0000","last_became_active":"2024-09-13T11:34:32.417122+0000","last_became_peered":"2024-09-13T11:34:32.417122+0000","last_unstale":"2024-09-13T11:35:03.799882+0000","last_undegraded":"2024-09-13T11:35:03.799882+0000","last_fullsized":"2024-09-13T11:35:03.799882+0000","mapping_epoch":123,"log_start":"101'344","ondisk_log_start":"101'344","created":30,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'346","last_scrub_stamp":"2024-09-13T11:34:23.260748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:23.260748+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":350,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:40.949534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326473,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":312,"num_read_kb":13587,"num_write":452,"num_write_kb":94522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":46662709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299281+0000","last_change":"2024-09-13T11:34:42.738098+0000","last_active":"2024-09-13T11:35:03.299281+0000","last_peered":"2024-09-13T11:35:03.299281+0000","last_clean":"2024-09-13T11:35:03.299281+0000","last_became_active":"2024-09-13T11:34:30.143931+0000","last_became_peered":"2024-09-13T11:34:30.143931+0000","last_unstale":"2024-09-13T11:35:03.299281+0000","last_undegraded":"2024-09-13T11:35:03.299281+0000","last_fullsized":"2024-09-13T11:35:03.299281+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:42.738023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:42.738023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:51.864419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"101'387","reported_seq":1045,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.075065+0000","last_change":"2024-09-13T11:35:04.075065+0000","last_active":"2024-09-13T11:35:04.075065+0000","last_peered":"2024-09-13T11:35:04.075065+0000","last_clean":"2024-09-13T11:35:04.075065+0000","last_became_active":"2024-09-13T11:34:35.426908+0000","last_became_peered":"2024-09-13T11:34:35.426908+0000","last_unstale":"2024-09-13T11:35:04.075065+0000","last_undegraded":"2024-09-13T11:35:04.075065+0000","last_fullsized":"2024-09-13T11:35:04.075065+0000","mapping_epoch":126,"log_start":"101'384","ondisk_log_start":"101'384","created":30,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'387","last_scrub_stamp":"2024-09-13T11:35:04.074861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:35:04.074861+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":399,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:05.177977+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":374,"num_read_kb":6471,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":86512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.981157+0000","last_change":"2024-09-13T11:34:51.125711+0000","last_active":"2024-09-13T11:35:03.981157+0000","last_peered":"2024-09-13T11:35:03.981157+0000","last_clean":"2024-09-13T11:35:03.981157+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:35:03.981157+0000","last_undegraded":"2024-09-13T11:35:03.981157+0000","last_fullsized":"2024-09-13T11:35:03.981157+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:51.125634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.125634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.787417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"101'422","reported_seq":938,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299314+0000","last_change":"2024-09-13T11:34:17.298237+0000","last_active":"2024-09-13T11:35:03.299314+0000","last_peered":"2024-09-13T11:35:03.299314+0000","last_clean":"2024-09-13T11:35:03.299314+0000","last_became_active":"2024-09-13T11:34:17.297909+0000","last_became_peered":"2024-09-13T11:34:17.297909+0000","last_unstale":"2024-09-13T11:35:03.299314+0000","last_undegraded":"2024-09-13T11:35:03.299314+0000","last_fullsized":"2024-09-13T11:35:03.299314+0000","mapping_epoch":110,"log_start":"101'418","ondisk_log_start":"101'418","created":30,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'422","last_scrub_stamp":"2024-09-13T11:34:11.938572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:11.938572+0000","objects_scrubbed":40,"log_size":4,"log_dups_size":433,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:32.609882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328092,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":400,"num_read_kb":5714,"num_write":543,"num_write_kb":103181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53636490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3],"acting":[13,12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":88,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:04.255512+0000","last_change":"2024-09-13T11:34:51.782015+0000","last_active":"2024-09-13T11:35:04.255512+0000","last_peered":"2024-09-13T11:35:04.255512+0000","last_clean":"2024-09-13T11:35:04.255512+0000","last_became_active":"2024-09-13T11:34:30.143993+0000","last_became_peered":"2024-09-13T11:34:30.143993+0000","last_unstale":"2024-09-13T11:35:04.255512+0000","last_undegraded":"2024-09-13T11:35:04.255512+0000","last_fullsized":"2024-09-13T11:35:04.255512+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:34:51.781912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.781912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:02.961286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'112","reported_seq":413,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-13T11:35:03.299291+0000","last_change":"2024-09-13T11:34:46.802627+0000","last_active":"2024-09-13T11:35:03.299291+0000","last_peered":"2024-09-13T11:35:03.299291+0000","last_clean":"2024-09-13T11:35:03.299291+0000","last_became_active":"2024-09-13T11:34:01.289125+0000","last_became_peered":"2024-09-13T11:34:01.289125+0000","last_unstale":"2024-09-13T11:35:03.299291+0000","last_undegraded":"2024-09-13T11:35:03.299291+0000","last_fullsized":"2024-09-13T11:35:03.299291+0000","mapping_epoch":100,"log_start":"25'110","ondisk_log_start":"25'110","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'112","last_scrub_stamp":"2024-09-13T11:34:46.802521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:03.312547+0000","last_clean_scrub_stamp":"2024-09-13T11:34:46.802521+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":110,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:03.688704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":112391362,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1439,"num_read_kb":35674,"num_write":1959,"num_write_kb":390122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":226892403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389038080,"data_stored":387751798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247849,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71552,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94300288,"statfs":{"total":96636764160,"available":96563494912,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247848,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84876,"kb_used_data":49532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286964,"statfs":{"total":96636764160,"available":96549851136,"internally_reserved":0,"allocated":50720768,"data_stored":49947521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247848,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58256,"kb_used_data":27072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313584,"statfs":{"total":96636764160,"available":96577110016,"internally_reserved":0,"allocated":27721728,"data_stored":27013733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280551,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1651040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67392,"kb_used_data":36208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304448,"statfs":{"total":96636764160,"available":96567754752,"internally_reserved":0,"allocated":37076992,"data_stored":36379481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32808,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339032,"statfs":{"total":96636764160,"available":96603168768,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":116,"seq":498216206346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52996,"kb_used_data":20272,"kb_used_omap":6,"kb_used_meta":32697,"kb_avail":94318844,"statfs":{"total":96636764160,"available":96582496256,"internally_reserved":0,"allocated":20758528,"data_stored":20062293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6811,"internal_metadata":33482085},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55248,"kb_used_data":24064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316592,"statfs":{"total":96636764160,"available":96580190208,"internally_reserved":0,"allocated":24641536,"data_stored":23979151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343204,"statfs":{"total":96636764160,"available":96607440896,"internally_reserved":0,"allocated":1650688,"data_stored":1053008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28628,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343212,"statfs":{"total":96636764160,"available":96607449088,"internally_reserved":0,"allocated":1642496,"data_stored":1045363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61000,"kb_used_data":29816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310840,"statfs":{"total":96636764160,"available":96574300160,"internally_reserved":0,"allocated":30531584,"data_stored":29829080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28648,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343192,"statfs":{"total":96636764160,"available":96607428608,"internally_reserved":0,"allocated":1662976,"data_stored":1060672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65148,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306692,"statfs":{"total":96636764160,"available":96570052608,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[2,3,4,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32800,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339040,"statfs":{"total":96636764160,"available":96603176960,"internally_reserved":0,"allocated":1654784,"data_stored":1053027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60988,"kb_used_data":29804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310852,"statfs":{"total":96636764160,"available":96574312448,"internally_reserved":0,"allocated":30519296,"data_stored":29821416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":108,"seq":463856467980,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86060,"kb_used_data":58076,"kb_used_omap":7,"kb_used_meta":27960,"kb_avail":94285780,"statfs":{"total":96636764160,"available":96548638720,"internally_reserved":0,"allocated":59469824,"data_stored":58715237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8160,"internal_metadata":28631072},"hb_peers":[1,2,3,5,6,7,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19107840,"data_stored":19009285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26083328,"data_stored":25968389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":57654565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":48463872,"data_stored":48296481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:35:07.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T11:35:08.037 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:08.038 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":156,"fsid":"b332d022-c5e6-4682-9d3d-2b6994cd0f20","created":"2024-09-13T11:31:50.491934+0000","modified":"2024-09-13T11:35:05.292010+0000","last_up_change":"2024-09-13T11:34:25.125921+0000","last_in_change":"2024-09-13T11:34:31.396117+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T11:32:02.947125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T11:32:05.120118+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-13T11:32:17.683275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":7.940000057220459,"score_stable":7.940000057220459,"optimal_score":0.62999999523162842,"raw_score_acting":5,"raw_score_stable":5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5a4d0d43-6562-441e-925e-4c992ee7b335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6805","nonce":2677506544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2677506544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2677506544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":2677506544}]},"public_addr":"172.21.15.38:6805/2677506544","cluster_addr":"172.21.15.38:6806/2677506544","heartbeat_back_addr":"172.21.15.38:6808/2677506544","heartbeat_front_addr":"172.21.15.38:6807/2677506544","state":["exists","up"]},{"osd":1,"uuid":"80665caa-6933-4f89-b454-128057e9ad92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":3561806747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":3561806747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":3561806747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":3561806747}]},"public_addr":"172.21.15.118:6808/3561806747","cluster_addr":"172.21.15.118:6809/3561806747","heartbeat_back_addr":"172.21.15.118:6811/3561806747","heartbeat_front_addr":"172.21.15.118:6810/3561806747","state":["exists","up"]},{"osd":2,"uuid":"eb99fd51-a064-4311-932e-8b6d555409af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":39,"up_from":116,"up_thru":0,"down_at":41,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":3302524927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":3302524927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":3302524927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":3302524927}]},"public_addr":"172.21.15.154:6800/3302524927","cluster_addr":"172.21.15.154:6801/3302524927","heartbeat_back_addr":"172.21.15.154:6803/3302524927","heartbeat_front_addr":"172.21.15.154:6802/3302524927","state":["exists","up"]},{"osd":3,"uuid":"3d840ed9-a410-4bcb-b0f0-70fafdf69842","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6804","nonce":2326505959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6805","nonce":2326505959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6807","nonce":2326505959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6806","nonce":2326505959}]},"public_addr":"172.21.15.204:6804/2326505959","cluster_addr":"172.21.15.204:6805/2326505959","heartbeat_back_addr":"172.21.15.204:6807/2326505959","heartbeat_front_addr":"172.21.15.204:6806/2326505959","state":["exists","up"]},{"osd":4,"uuid":"0c01b7d7-1309-40fa-913a-97d2a8499293","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6801","nonce":3107658120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":3107658120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":3107658120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":3107658120}]},"public_addr":"172.21.15.38:6801/3107658120","cluster_addr":"172.21.15.38:6802/3107658120","heartbeat_back_addr":"172.21.15.38:6804/3107658120","heartbeat_front_addr":"172.21.15.38:6803/3107658120","state":["exists","up"]},{"osd":5,"uuid":"b4cec928-1764-4fe0-8c5d-870d8a0c15c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3800782738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":3800782738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":3800782738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":3800782738}]},"public_addr":"172.21.15.118:6800/3800782738","cluster_addr":"172.21.15.118:6801/3800782738","heartbeat_back_addr":"172.21.15.118:6803/3800782738","heartbeat_front_addr":"172.21.15.118:6802/3800782738","state":["exists","up"]},{"osd":6,"uuid":"89f6d810-2afc-42b0-b989-052218b6887c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3986362716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3986362716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3986362716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3986362716}]},"public_addr":"172.21.15.154:6808/3986362716","cluster_addr":"172.21.15.154:6809/3986362716","heartbeat_back_addr":"172.21.15.154:6811/3986362716","heartbeat_front_addr":"172.21.15.154:6810/3986362716","state":["exists","up"]},{"osd":7,"uuid":"ccfee49f-f394-4de0-8e30-853f38e31232","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6808","nonce":1978982202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6809","nonce":1978982202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6811","nonce":1978982202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6810","nonce":1978982202}]},"public_addr":"172.21.15.204:6808/1978982202","cluster_addr":"172.21.15.204:6809/1978982202","heartbeat_back_addr":"172.21.15.204:6811/1978982202","heartbeat_front_addr":"172.21.15.204:6810/1978982202","state":["exists","up"]},{"osd":8,"uuid":"03b1c505-1b92-4daa-9da4-c254be2ba792","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3608171775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3608171775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3608171775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6815","nonce":3608171775}]},"public_addr":"172.21.15.38:6813/3608171775","cluster_addr":"172.21.15.38:6814/3608171775","heartbeat_back_addr":"172.21.15.38:6816/3608171775","heartbeat_front_addr":"172.21.15.38:6815/3608171775","state":["exists","up"]},{"osd":9,"uuid":"8fbee7a0-2ee6-4551-bf06-4e013e413fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2580659844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":2580659844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":2580659844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2580659844}]},"public_addr":"172.21.15.118:6804/2580659844","cluster_addr":"172.21.15.118:6805/2580659844","heartbeat_back_addr":"172.21.15.118:6807/2580659844","heartbeat_front_addr":"172.21.15.118:6806/2580659844","state":["exists","up"]},{"osd":10,"uuid":"0808cb41-f62e-49b0-935d-616bea1a4db5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2352891695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":2352891695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":2352891695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2352891695}]},"public_addr":"172.21.15.154:6812/2352891695","cluster_addr":"172.21.15.154:6813/2352891695","heartbeat_back_addr":"172.21.15.154:6815/2352891695","heartbeat_front_addr":"172.21.15.154:6814/2352891695","state":["exists","up"]},{"osd":11,"uuid":"427dd574-f2a8-49e5-a32a-cfda023a514a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6800","nonce":2304836889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6801","nonce":2304836889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6803","nonce":2304836889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6802","nonce":2304836889}]},"public_addr":"172.21.15.204:6800/2304836889","cluster_addr":"172.21.15.204:6801/2304836889","heartbeat_back_addr":"172.21.15.204:6803/2304836889","heartbeat_front_addr":"172.21.15.204:6802/2304836889","state":["exists","up"]},{"osd":12,"uuid":"17942689-1130-4b6b-9f98-1605a03731b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":58,"up_from":108,"up_thru":123,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6809","nonce":3237768415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3237768415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3237768415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6811","nonce":3237768415}]},"public_addr":"172.21.15.38:6809/3237768415","cluster_addr":"172.21.15.38:6810/3237768415","heartbeat_back_addr":"172.21.15.38:6812/3237768415","heartbeat_front_addr":"172.21.15.38:6811/3237768415","state":["exists","up"]},{"osd":13,"uuid":"62161cab-6d14-4571-971d-217601e38fbc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":3727425991}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":3727425991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":3727425991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":3727425991}]},"public_addr":"172.21.15.118:6812/3727425991","cluster_addr":"172.21.15.118:6813/3727425991","heartbeat_back_addr":"172.21.15.118:6815/3727425991","heartbeat_front_addr":"172.21.15.118:6814/3727425991","state":["exists","up"]},{"osd":14,"uuid":"559ef141-27db-452d-9577-a6135421f64f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":538351706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":538351706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":538351706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":538351706}]},"public_addr":"172.21.15.154:6804/538351706","cluster_addr":"172.21.15.154:6805/538351706","heartbeat_back_addr":"172.21.15.154:6807/538351706","heartbeat_front_addr":"172.21.15.154:6806/538351706","state":["exists","up"]},{"osd":15,"uuid":"98f9ceba-79df-410a-9788-393057e79c30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":100,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6812","nonce":501019151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6813","nonce":501019151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6815","nonce":501019151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.204:6814","nonce":501019151}]},"public_addr":"172.21.15.204:6812/501019151","cluster_addr":"172.21.15.204:6813/501019151","heartbeat_back_addr":"172.21.15.204:6815/501019151","heartbeat_front_addr":"172.21.15.204:6814/501019151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:22.298728+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:29.906048+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:24.237851+0000","dead_epoch":41},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:28.159438+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:27.889932+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:21.167035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:28.767016+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:38.597935+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:26.703342+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:37.877924+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:34.475326+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:33.370280+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:13.428589+0000","dead_epoch":59},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:38.011256+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:46.313159+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T11:34:09.456441+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.7","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T11:35:09.052 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T11:35:09.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:09.194 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:09.194 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:09.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T11:35:09.509 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T11:35:09.522 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T11:35:09.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:09.649 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:09.650 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:09.650 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:09.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T11:35:09.962 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T11:35:09.974 INFO:tasks.ceph:Scrubbing osd.8 2024-09-13T11:35:09.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:10.102 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:10.103 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:10.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-13T11:35:10.426 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 8 to deep-scrub 2024-09-13T11:35:10.439 INFO:tasks.ceph:Scrubbing osd.12 2024-09-13T11:35:10.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:10.568 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:10.568 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:10.568 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:10.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-13T11:35:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 12 to deep-scrub 2024-09-13T11:35:10.892 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T11:35:10.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:11.021 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:11.022 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:11.022 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:11.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T11:35:11.333 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T11:35:11.346 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T11:35:11.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:11.482 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:11.482 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:11.482 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:11.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T11:35:11.795 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T11:35:11.807 INFO:tasks.ceph:Scrubbing osd.9 2024-09-13T11:35:11.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:11.937 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:11.937 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:11.937 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:11.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-13T11:35:12.246 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 9 to deep-scrub 2024-09-13T11:35:12.259 INFO:tasks.ceph:Scrubbing osd.13 2024-09-13T11:35:12.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:12.389 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:12.390 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:12.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-13T11:35:12.699 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 13 to deep-scrub 2024-09-13T11:35:12.713 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T11:35:12.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:12.842 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:12.842 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:12.853 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T11:35:13.153 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T11:35:13.166 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T11:35:13.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:13.294 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:13.295 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:13.295 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:13.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T11:35:13.606 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T11:35:13.619 INFO:tasks.ceph:Scrubbing osd.10 2024-09-13T11:35:13.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:13.748 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:13.748 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:13.748 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:13.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-13T11:35:14.062 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 10 to deep-scrub 2024-09-13T11:35:14.075 INFO:tasks.ceph:Scrubbing osd.14 2024-09-13T11:35:14.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:14.205 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:14.205 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:14.205 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:14.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-13T11:35:14.516 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 14 to deep-scrub 2024-09-13T11:35:14.530 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T11:35:14.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:14.661 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:14.661 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:14.661 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:14.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T11:35:14.974 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T11:35:14.987 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T11:35:14.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:15.117 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:15.117 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:15.117 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:15.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T11:35:15.428 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T11:35:15.442 INFO:tasks.ceph:Scrubbing osd.11 2024-09-13T11:35:15.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:15.571 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:15.572 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:15.572 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:15.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-13T11:35:15.885 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 11 to deep-scrub 2024-09-13T11:35:15.899 INFO:tasks.ceph:Scrubbing osd.15 2024-09-13T11:35:15.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-13T11:35:16.028 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-09-13T11:35:16.028 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T11:35:16.028 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-09-13T11:35:16.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-13T11:35:16.342 INFO:teuthology.orchestra.run.smithi038.stderr:instructed osd(s) 15 to deep-scrub 2024-09-13T11:35:16.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:35:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:16.665 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:35:16.678 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-13T11:35:14.969515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112981749,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":1525,"num_read_kb":35747,"num_write":2106,"num_write_kb":392796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":228073177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":852916,"kb_used_data":351464,"kb_used_omap":36,"kb_used_meta":501147,"kb_avail":1509096524,"statfs":{"total":1546188226560,"available":1545314840576,"internally_reserved":0,"allocated":359899136,"data_stored":349079391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37303,"internal_metadata":513175113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002099"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":312,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325611+0000","last_change":"2024-09-13T11:34:55.901795+0000","last_active":"2024-09-13T11:35:05.325611+0000","last_peered":"2024-09-13T11:35:05.325611+0000","last_clean":"2024-09-13T11:35:05.325611+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:35:05.325611+0000","last_undegraded":"2024-09-13T11:35:05.325611+0000","last_fullsized":"2024-09-13T11:35:05.325611+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:55.901697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:55.901697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:25.291243+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":289,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.296989+0000","last_change":"2024-09-13T11:35:00.411948+0000","last_active":"2024-09-13T11:35:05.296989+0000","last_peered":"2024-09-13T11:35:05.296989+0000","last_clean":"2024-09-13T11:35:05.296989+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:35:05.296989+0000","last_undegraded":"2024-09-13T11:35:05.296989+0000","last_fullsized":"2024-09-13T11:35:05.296989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:00.411854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:35:00.411854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:19.314223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":301,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:11.417409+0000","last_change":"2024-09-13T11:35:11.417409+0000","last_active":"2024-09-13T11:35:11.417409+0000","last_peered":"2024-09-13T11:35:11.417409+0000","last_clean":"2024-09-13T11:35:11.417409+0000","last_became_active":"2024-09-13T11:34:26.221059+0000","last_became_peered":"2024-09-13T11:34:26.221059+0000","last_unstale":"2024-09-13T11:35:11.417409+0000","last_undegraded":"2024-09-13T11:35:11.417409+0000","last_fullsized":"2024-09-13T11:35:11.417409+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:11.417343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:35:11.417343+0000","last_clean_scrub_stamp":"2024-09-13T11:35:11.417343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:23.680492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":294,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325287+0000","last_change":"2024-09-13T11:34:18.999684+0000","last_active":"2024-09-13T11:35:05.325287+0000","last_peered":"2024-09-13T11:35:05.325287+0000","last_clean":"2024-09-13T11:35:05.325287+0000","last_became_active":"2024-09-13T11:34:15.290566+0000","last_became_peered":"2024-09-13T11:34:15.290566+0000","last_unstale":"2024-09-13T11:35:05.325287+0000","last_undegraded":"2024-09-13T11:35:05.325287+0000","last_fullsized":"2024-09-13T11:35:05.325287+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:18.999603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:18.999603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:37.998553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"101'356","reported_seq":873,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325186+0000","last_change":"2024-09-13T11:34:37.797878+0000","last_active":"2024-09-13T11:35:05.325186+0000","last_peered":"2024-09-13T11:35:05.325186+0000","last_clean":"2024-09-13T11:35:05.325186+0000","last_became_active":"2024-09-13T11:34:28.142159+0000","last_became_peered":"2024-09-13T11:34:28.142159+0000","last_unstale":"2024-09-13T11:35:05.325186+0000","last_undegraded":"2024-09-13T11:35:05.325186+0000","last_fullsized":"2024-09-13T11:35:05.325186+0000","mapping_epoch":118,"log_start":"101'353","ondisk_log_start":"101'353","created":30,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'356","last_scrub_stamp":"2024-09-13T11:34:37.797738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:37.797738+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":362,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:55.713319+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968389,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":353,"num_read_kb":9902,"num_write":469,"num_write_kb":89073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":40081204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2],"acting":[13,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":315,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325928+0000","last_change":"2024-09-13T11:34:38.032423+0000","last_active":"2024-09-13T11:35:05.325928+0000","last_peered":"2024-09-13T11:35:05.325928+0000","last_clean":"2024-09-13T11:35:05.325928+0000","last_became_active":"2024-09-13T11:34:30.146538+0000","last_became_peered":"2024-09-13T11:34:30.146538+0000","last_unstale":"2024-09-13T11:35:05.325928+0000","last_undegraded":"2024-09-13T11:35:05.325928+0000","last_fullsized":"2024-09-13T11:35:05.325928+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-13T11:34:38.032294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:38.032294+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.425169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"156'395","reported_seq":947,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:11.471121+0000","last_change":"2024-09-13T11:35:11.471121+0000","last_active":"2024-09-13T11:35:11.471121+0000","last_peered":"2024-09-13T11:35:11.471121+0000","last_clean":"2024-09-13T11:35:11.471121+0000","last_became_active":"2024-09-13T11:34:32.417122+0000","last_became_peered":"2024-09-13T11:34:32.417122+0000","last_unstale":"2024-09-13T11:35:11.471121+0000","last_undegraded":"2024-09-13T11:35:11.471121+0000","last_fullsized":"2024-09-13T11:35:11.471121+0000","mapping_epoch":123,"log_start":"156'370","ondisk_log_start":"156'370","created":30,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'395","last_scrub_stamp":"2024-09-13T11:35:11.470951+0000","last_deep_scrub":"156'395","last_deep_scrub_stamp":"2024-09-13T11:35:11.470951+0000","last_clean_scrub_stamp":"2024-09-13T11:35:11.470951+0000","objects_scrubbed":49,"log_size":25,"log_dups_size":376,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:20.322244+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326473,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":312,"num_read_kb":13587,"num_write":452,"num_write_kb":94522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":46662709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":137,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325138+0000","last_change":"2024-09-13T11:34:42.738098+0000","last_active":"2024-09-13T11:35:05.325138+0000","last_peered":"2024-09-13T11:35:05.325138+0000","last_clean":"2024-09-13T11:35:05.325138+0000","last_became_active":"2024-09-13T11:34:30.143931+0000","last_became_peered":"2024-09-13T11:34:30.143931+0000","last_unstale":"2024-09-13T11:35:05.325138+0000","last_undegraded":"2024-09-13T11:35:05.325138+0000","last_fullsized":"2024-09-13T11:35:05.325138+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:42.738023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:42.738023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:51.864419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"101'387","reported_seq":1047,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325274+0000","last_change":"2024-09-13T11:35:04.075065+0000","last_active":"2024-09-13T11:35:05.325274+0000","last_peered":"2024-09-13T11:35:05.325274+0000","last_clean":"2024-09-13T11:35:05.325274+0000","last_became_active":"2024-09-13T11:34:35.426908+0000","last_became_peered":"2024-09-13T11:34:35.426908+0000","last_unstale":"2024-09-13T11:35:05.325274+0000","last_undegraded":"2024-09-13T11:35:05.325274+0000","last_fullsized":"2024-09-13T11:35:05.325274+0000","mapping_epoch":126,"log_start":"101'384","ondisk_log_start":"101'384","created":30,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'387","last_scrub_stamp":"2024-09-13T11:35:04.074861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:35:04.074861+0000","objects_scrubbed":54,"log_size":3,"log_dups_size":399,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:05.177977+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":374,"num_read_kb":6471,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":86512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":291,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325856+0000","last_change":"2024-09-13T11:34:51.125711+0000","last_active":"2024-09-13T11:35:05.325856+0000","last_peered":"2024-09-13T11:35:05.325856+0000","last_clean":"2024-09-13T11:35:05.325856+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:35:05.325856+0000","last_undegraded":"2024-09-13T11:35:05.325856+0000","last_fullsized":"2024-09-13T11:35:05.325856+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:34:51.125634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.125634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:58.787417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"101'422","reported_seq":940,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.325172+0000","last_change":"2024-09-13T11:34:17.298237+0000","last_active":"2024-09-13T11:35:05.325172+0000","last_peered":"2024-09-13T11:35:05.325172+0000","last_clean":"2024-09-13T11:35:05.325172+0000","last_became_active":"2024-09-13T11:34:17.297909+0000","last_became_peered":"2024-09-13T11:34:17.297909+0000","last_unstale":"2024-09-13T11:35:05.325172+0000","last_undegraded":"2024-09-13T11:35:05.325172+0000","last_fullsized":"2024-09-13T11:35:05.325172+0000","mapping_epoch":110,"log_start":"101'418","ondisk_log_start":"101'418","created":30,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'422","last_scrub_stamp":"2024-09-13T11:34:11.938572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:18.407384+0000","last_clean_scrub_stamp":"2024-09-13T11:34:11.938572+0000","objects_scrubbed":40,"log_size":4,"log_dups_size":433,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:35:32.609882+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328092,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":400,"num_read_kb":5714,"num_write":543,"num_write_kb":103181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53636490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3],"acting":[13,12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":90,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:05.501149+0000","last_change":"2024-09-13T11:34:51.782015+0000","last_active":"2024-09-13T11:35:05.501149+0000","last_peered":"2024-09-13T11:35:05.501149+0000","last_clean":"2024-09-13T11:35:05.501149+0000","last_became_active":"2024-09-13T11:34:30.143993+0000","last_became_peered":"2024-09-13T11:34:30.143993+0000","last_unstale":"2024-09-13T11:35:05.501149+0000","last_undegraded":"2024-09-13T11:35:05.501149+0000","last_fullsized":"2024-09-13T11:35:05.501149+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-13T11:34:51.781912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:32:05.330509+0000","last_clean_scrub_stamp":"2024-09-13T11:34:51.781912+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:02.961286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"156'114","reported_seq":425,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-09-13T11:35:13.509097+0000","last_change":"2024-09-13T11:35:13.509097+0000","last_active":"2024-09-13T11:35:13.509097+0000","last_peered":"2024-09-13T11:35:13.509097+0000","last_clean":"2024-09-13T11:35:13.509097+0000","last_became_active":"2024-09-13T11:34:01.289125+0000","last_became_peered":"2024-09-13T11:34:01.289125+0000","last_unstale":"2024-09-13T11:35:13.509097+0000","last_undegraded":"2024-09-13T11:35:13.509097+0000","last_fullsized":"2024-09-13T11:35:13.509097+0000","mapping_epoch":100,"log_start":"25'111","ondisk_log_start":"25'111","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'114","last_scrub_stamp":"2024-09-13T11:35:13.508995+0000","last_deep_scrub":"156'114","last_deep_scrub_stamp":"2024-09-13T11:35:13.508995+0000","last_clean_scrub_stamp":"2024-09-13T11:35:13.508995+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":111,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:38.806227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":112391362,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1439,"num_read_kb":35674,"num_write":1959,"num_write_kb":390122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":226892403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389038080,"data_stored":387751798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":12,"acting":12,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71576,"kb_used_data":36232,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94300264,"statfs":{"total":96636764160,"available":96563470336,"internally_reserved":0,"allocated":37101568,"data_stored":36394801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247850,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84888,"kb_used_data":49544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286952,"statfs":{"total":96636764160,"available":96549838848,"internally_reserved":0,"allocated":50733056,"data_stored":49955177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58292,"kb_used_data":27108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313548,"statfs":{"total":96636764160,"available":96577073152,"internally_reserved":0,"allocated":27758592,"data_stored":27036717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37560,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334280,"statfs":{"total":96636764160,"available":96598302720,"internally_reserved":0,"allocated":2269184,"data_stored":1658696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67416,"kb_used_data":36232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304424,"statfs":{"total":96636764160,"available":96567730176,"internally_reserved":0,"allocated":37101568,"data_stored":36394801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280553,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32820,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339020,"statfs":{"total":96636764160,"available":96603156480,"internally_reserved":0,"allocated":1675264,"data_stored":1068328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":116,"seq":498216206348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53028,"kb_used_data":20296,"kb_used_omap":6,"kb_used_meta":32697,"kb_avail":94318812,"statfs":{"total":96636764160,"available":96582463488,"internally_reserved":0,"allocated":20783104,"data_stored":20077613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6811,"internal_metadata":33482085},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280554,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55260,"kb_used_data":24076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316580,"statfs":{"total":96636764160,"available":96580177920,"internally_reserved":0,"allocated":24653824,"data_stored":23986807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":1675264,"data_stored":1068328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280553,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28664,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343176,"statfs":{"total":96636764160,"available":96607412224,"internally_reserved":0,"allocated":1679360,"data_stored":1068347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280554,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61012,"kb_used_data":29828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310828,"statfs":{"total":96636764160,"available":96574287872,"internally_reserved":0,"allocated":30543872,"data_stored":29836736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280554,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":1675264,"data_stored":1068328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65172,"kb_used_data":29828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306668,"statfs":{"total":96636764160,"available":96570028032,"internally_reserved":0,"allocated":30543872,"data_stored":29836736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[2,3,4,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280554,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32824,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339016,"statfs":{"total":96636764160,"available":96603152384,"internally_reserved":0,"allocated":1679360,"data_stored":1068347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313258,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61012,"kb_used_data":29828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310828,"statfs":{"total":96636764160,"available":96574287872,"internally_reserved":0,"allocated":30543872,"data_stored":29836736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":108,"seq":463856467982,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86072,"kb_used_data":58088,"kb_used_omap":7,"kb_used_meta":27960,"kb_avail":94285768,"statfs":{"total":96636764160,"available":96548626432,"internally_reserved":0,"allocated":59482112,"data_stored":58722893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8160,"internal_metadata":28631072},"hb_peers":[1,2,3,5,6,7,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19107840,"data_stored":19009285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26083328,"data_stored":25968389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":57654565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":48463872,"data_stored":48296481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:35:16.682 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T11:34:55.901697+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=55, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T11:35:00.411854+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=0, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T11:34:18.999603+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=18, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-13T11:34:37.797738+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=37, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T11:34:38.032294+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=38, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T11:34:42.738023+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=42, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-13T11:35:04.074861+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=4, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T11:34:51.125634+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=51, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.683 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-13T11:34:11.938572+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.684 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T11:34:51.781912+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=34, tm_sec=51, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=11, tm_min=35, tm_sec=8, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T11:35:16.684 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T11:35:36.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T11:35:37.002 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-09-13T11:35:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-09-13T11:35:37.014 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-13T11:35:36.974820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112981749,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":0,"num_read":1525,"num_read_kb":35747,"num_write":2106,"num_write_kb":392796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":228073177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":853268,"kb_used_data":351824,"kb_used_omap":36,"kb_used_meta":501147,"kb_avail":1509096172,"statfs":{"total":1546188226560,"available":1545314480128,"internally_reserved":0,"allocated":360267776,"data_stored":349310121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37303,"internal_metadata":513175113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.669075"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":326,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.787263+0000","last_change":"2024-09-13T11:35:17.000968+0000","last_active":"2024-09-13T11:35:32.787263+0000","last_peered":"2024-09-13T11:35:32.787263+0000","last_clean":"2024-09-13T11:35:32.787263+0000","last_became_active":"2024-09-13T11:33:58.558897+0000","last_became_peered":"2024-09-13T11:33:58.558897+0000","last_unstale":"2024-09-13T11:35:32.787263+0000","last_undegraded":"2024-09-13T11:35:32.787263+0000","last_fullsized":"2024-09-13T11:35:32.787263+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:17.000883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:35:17.000883+0000","last_clean_scrub_stamp":"2024-09-13T11:35:17.000883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:31.395115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00046472499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,0],"acting":[15,13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":301,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.286538+0000","last_change":"2024-09-13T11:35:15.327509+0000","last_active":"2024-09-13T11:35:32.286538+0000","last_peered":"2024-09-13T11:35:32.286538+0000","last_clean":"2024-09-13T11:35:32.286538+0000","last_became_active":"2024-09-13T11:32:06.608544+0000","last_became_peered":"2024-09-13T11:32:06.608544+0000","last_unstale":"2024-09-13T11:35:32.286538+0000","last_undegraded":"2024-09-13T11:35:32.286538+0000","last_fullsized":"2024-09-13T11:35:32.286538+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:15.327425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:35:15.327425+0000","last_clean_scrub_stamp":"2024-09-13T11:35:15.327425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:19.213805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054460800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,4],"acting":[14,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":305,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T11:35:30.304381+0000","last_change":"2024-09-13T11:35:11.417409+0000","last_active":"2024-09-13T11:35:30.304381+0000","last_peered":"2024-09-13T11:35:30.304381+0000","last_clean":"2024-09-13T11:35:30.304381+0000","last_became_active":"2024-09-13T11:34:26.221059+0000","last_became_peered":"2024-09-13T11:34:26.221059+0000","last_unstale":"2024-09-13T11:35:30.304381+0000","last_undegraded":"2024-09-13T11:35:30.304381+0000","last_fullsized":"2024-09-13T11:35:30.304381+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:11.417343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:35:11.417343+0000","last_clean_scrub_stamp":"2024-09-13T11:35:11.417343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:23.680492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035337999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,6],"acting":[8,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":306,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T11:35:30.306847+0000","last_change":"2024-09-13T11:35:15.181417+0000","last_active":"2024-09-13T11:35:30.306847+0000","last_peered":"2024-09-13T11:35:30.306847+0000","last_clean":"2024-09-13T11:35:30.306847+0000","last_became_active":"2024-09-13T11:34:15.290566+0000","last_became_peered":"2024-09-13T11:34:15.290566+0000","last_unstale":"2024-09-13T11:35:30.306847+0000","last_undegraded":"2024-09-13T11:35:30.306847+0000","last_fullsized":"2024-09-13T11:35:30.306847+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:15.181306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:35:15.181306+0000","last_clean_scrub_stamp":"2024-09-13T11:35:15.181306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:23.576555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056170199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,12,10],"acting":[6,12,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"156'410","reported_seq":942,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.412220+0000","last_change":"2024-09-13T11:35:15.625611+0000","last_active":"2024-09-13T11:35:32.412220+0000","last_peered":"2024-09-13T11:35:32.412220+0000","last_clean":"2024-09-13T11:35:32.412220+0000","last_became_active":"2024-09-13T11:34:28.142159+0000","last_became_peered":"2024-09-13T11:34:28.142159+0000","last_unstale":"2024-09-13T11:35:32.412220+0000","last_undegraded":"2024-09-13T11:35:32.412220+0000","last_fullsized":"2024-09-13T11:35:32.412220+0000","mapping_epoch":118,"log_start":"156'405","ondisk_log_start":"156'405","created":30,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'410","last_scrub_stamp":"2024-09-13T11:35:15.625527+0000","last_deep_scrub":"156'410","last_deep_scrub_stamp":"2024-09-13T11:35:15.625527+0000","last_clean_scrub_stamp":"2024-09-13T11:35:15.625527+0000","objects_scrubbed":54,"log_size":5,"log_dups_size":414,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:15.839537+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25968389,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":353,"num_read_kb":9902,"num_write":469,"num_write_kb":89073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":40081204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,2],"acting":[13,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"156'3","reported_seq":326,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T11:35:27.398201+0000","last_change":"2024-09-13T11:35:16.161215+0000","last_active":"2024-09-13T11:35:27.398201+0000","last_peered":"2024-09-13T11:35:27.398201+0000","last_clean":"2024-09-13T11:35:27.398201+0000","last_became_active":"2024-09-13T11:34:30.146538+0000","last_became_peered":"2024-09-13T11:34:30.146538+0000","last_unstale":"2024-09-13T11:35:27.398201+0000","last_undegraded":"2024-09-13T11:35:27.398201+0000","last_fullsized":"2024-09-13T11:35:27.398201+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'3","last_scrub_stamp":"2024-09-13T11:35:16.161146+0000","last_deep_scrub":"156'3","last_deep_scrub_stamp":"2024-09-13T11:35:16.161146+0000","last_clean_scrub_stamp":"2024-09-13T11:35:16.161146+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:35.719731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003798984,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,5],"acting":[3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"156'395","reported_seq":952,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.412053+0000","last_change":"2024-09-13T11:35:11.471121+0000","last_active":"2024-09-13T11:35:32.412053+0000","last_peered":"2024-09-13T11:35:32.412053+0000","last_clean":"2024-09-13T11:35:32.412053+0000","last_became_active":"2024-09-13T11:34:32.417122+0000","last_became_peered":"2024-09-13T11:34:32.417122+0000","last_unstale":"2024-09-13T11:35:32.412053+0000","last_undegraded":"2024-09-13T11:35:32.412053+0000","last_fullsized":"2024-09-13T11:35:32.412053+0000","mapping_epoch":123,"log_start":"156'370","ondisk_log_start":"156'370","created":30,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'395","last_scrub_stamp":"2024-09-13T11:35:11.470951+0000","last_deep_scrub":"156'395","last_deep_scrub_stamp":"2024-09-13T11:35:11.470951+0000","last_clean_scrub_stamp":"2024-09-13T11:35:11.470951+0000","objects_scrubbed":49,"log_size":25,"log_dups_size":376,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:20.322244+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35326473,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":312,"num_read_kb":13587,"num_write":452,"num_write_kb":94522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":46662709,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.412234+0000","last_change":"2024-09-13T11:35:14.530136+0000","last_active":"2024-09-13T11:35:32.412234+0000","last_peered":"2024-09-13T11:35:32.412234+0000","last_clean":"2024-09-13T11:35:32.412234+0000","last_became_active":"2024-09-13T11:34:30.143931+0000","last_became_peered":"2024-09-13T11:34:30.143931+0000","last_unstale":"2024-09-13T11:35:32.412234+0000","last_undegraded":"2024-09-13T11:35:32.412234+0000","last_fullsized":"2024-09-13T11:35:32.412234+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:14.530052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:35:14.530052+0000","last_clean_scrub_stamp":"2024-09-13T11:35:14.530052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:39.007075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044991999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5,14],"acting":[13,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"156'441","reported_seq":1115,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T11:35:30.306848+0000","last_change":"2024-09-13T11:35:14.268990+0000","last_active":"2024-09-13T11:35:30.306848+0000","last_peered":"2024-09-13T11:35:30.306848+0000","last_clean":"2024-09-13T11:35:30.306848+0000","last_became_active":"2024-09-13T11:34:35.426908+0000","last_became_peered":"2024-09-13T11:34:35.426908+0000","last_unstale":"2024-09-13T11:35:30.306848+0000","last_undegraded":"2024-09-13T11:35:30.306848+0000","last_fullsized":"2024-09-13T11:35:30.306848+0000","mapping_epoch":126,"log_start":"156'436","ondisk_log_start":"156'436","created":30,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'441","last_scrub_stamp":"2024-09-13T11:35:14.268919+0000","last_deep_scrub":"156'441","last_deep_scrub_stamp":"2024-09-13T11:35:14.268919+0000","last_clean_scrub_stamp":"2024-09-13T11:35:14.268919+0000","objects_scrubbed":54,"log_size":5,"log_dups_size":451,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:37.533624+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28768408,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":374,"num_read_kb":6471,"num_write":495,"num_write_kb":103346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":86512000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,9],"acting":[6,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":301,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-13T11:35:27.398171+0000","last_change":"2024-09-13T11:35:15.116475+0000","last_active":"2024-09-13T11:35:27.398171+0000","last_peered":"2024-09-13T11:35:27.398171+0000","last_clean":"2024-09-13T11:35:27.398171+0000","last_became_active":"2024-09-13T11:32:06.608620+0000","last_became_peered":"2024-09-13T11:32:06.608620+0000","last_unstale":"2024-09-13T11:35:27.398171+0000","last_undegraded":"2024-09-13T11:35:27.398171+0000","last_fullsized":"2024-09-13T11:35:27.398171+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T11:35:15.116383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T11:35:15.116383+0000","last_clean_scrub_stamp":"2024-09-13T11:35:15.116383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:30.864588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041044299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,8],"acting":[3,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"156'462","reported_seq":994,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.412273+0000","last_change":"2024-09-13T11:35:16.635339+0000","last_active":"2024-09-13T11:35:32.412273+0000","last_peered":"2024-09-13T11:35:32.412273+0000","last_clean":"2024-09-13T11:35:32.412273+0000","last_became_active":"2024-09-13T11:34:17.297909+0000","last_became_peered":"2024-09-13T11:34:17.297909+0000","last_unstale":"2024-09-13T11:35:32.412273+0000","last_undegraded":"2024-09-13T11:35:32.412273+0000","last_fullsized":"2024-09-13T11:35:32.412273+0000","mapping_epoch":110,"log_start":"156'446","ondisk_log_start":"156'446","created":30,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'462","last_scrub_stamp":"2024-09-13T11:35:16.635236+0000","last_deep_scrub":"156'462","last_deep_scrub_stamp":"2024-09-13T11:35:16.635236+0000","last_clean_scrub_stamp":"2024-09-13T11:35:16.635236+0000","objects_scrubbed":40,"log_size":16,"log_dups_size":461,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:34.907727+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328092,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":400,"num_read_kb":5714,"num_write":543,"num_write_kb":103181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53636490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,3],"acting":[13,12,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"156'2","reported_seq":103,"reported_epoch":158,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.259460+0000","last_change":"2024-09-13T11:35:12.694161+0000","last_active":"2024-09-13T11:35:32.259460+0000","last_peered":"2024-09-13T11:35:32.259460+0000","last_clean":"2024-09-13T11:35:32.259460+0000","last_became_active":"2024-09-13T11:34:30.143993+0000","last_became_peered":"2024-09-13T11:34:30.143993+0000","last_unstale":"2024-09-13T11:35:32.259460+0000","last_undegraded":"2024-09-13T11:35:32.259460+0000","last_fullsized":"2024-09-13T11:35:32.259460+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'2","last_scrub_stamp":"2024-09-13T11:35:12.694062+0000","last_deep_scrub":"156'2","last_deep_scrub_stamp":"2024-09-13T11:35:12.694062+0000","last_clean_scrub_stamp":"2024-09-13T11:35:12.694062+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:41.284926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15,13],"acting":[5,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"156'114","reported_seq":430,"reported_epoch":159,"state":"active+clean","last_fresh":"2024-09-13T11:35:32.412220+0000","last_change":"2024-09-13T11:35:13.509097+0000","last_active":"2024-09-13T11:35:32.412220+0000","last_peered":"2024-09-13T11:35:32.412220+0000","last_clean":"2024-09-13T11:35:32.412220+0000","last_became_active":"2024-09-13T11:34:01.289125+0000","last_became_peered":"2024-09-13T11:34:01.289125+0000","last_unstale":"2024-09-13T11:35:32.412220+0000","last_undegraded":"2024-09-13T11:35:32.412220+0000","last_fullsized":"2024-09-13T11:35:32.412220+0000","mapping_epoch":100,"log_start":"25'111","ondisk_log_start":"25'111","created":23,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"156'114","last_scrub_stamp":"2024-09-13T11:35:13.508995+0000","last_deep_scrub":"156'114","last_deep_scrub_stamp":"2024-09-13T11:35:13.508995+0000","last_clean_scrub_stamp":"2024-09-13T11:35:13.508995+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":111,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-13T11:36:38.806227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0,3],"acting":[13,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":112391362,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1439,"num_read_kb":35674,"num_write":1959,"num_write_kb":390122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":226892403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":389038080,"data_stored":387751798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":12,"acting":12,"num_store_stats":12},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71612,"kb_used_data":36268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94300228,"statfs":{"total":96636764160,"available":96563433472,"internally_reserved":0,"allocated":37138432,"data_stored":36417869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84924,"kb_used_data":49580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286916,"statfs":{"total":96636764160,"available":96549801984,"internally_reserved":0,"allocated":50769920,"data_stored":49978245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,7,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":23,"seq":98784247854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58328,"kb_used_data":27144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94313512,"statfs":{"total":96636764160,"available":96577036288,"internally_reserved":0,"allocated":27795456,"data_stored":27059785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,4,6,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37584,"kb_used_data":2240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334256,"statfs":{"total":96636764160,"available":96598278144,"internally_reserved":0,"allocated":2293760,"data_stored":1674058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67428,"kb_used_data":36244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304412,"statfs":{"total":96636764160,"available":96567717888,"internally_reserved":0,"allocated":37113856,"data_stored":36402507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":1687552,"data_stored":1076034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":116,"seq":498216206352,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53056,"kb_used_data":20332,"kb_used_omap":6,"kb_used_meta":32697,"kb_avail":94318784,"statfs":{"total":96636764160,"available":96582434816,"internally_reserved":0,"allocated":20819968,"data_stored":20100681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6811,"internal_metadata":33482085},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55272,"kb_used_data":24088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316568,"statfs":{"total":96636764160,"available":96580165632,"internally_reserved":0,"allocated":24666112,"data_stored":23994513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28672,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343168,"statfs":{"total":96636764160,"available":96607404032,"internally_reserved":0,"allocated":1687552,"data_stored":1076034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28676,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343164,"statfs":{"total":96636764160,"available":96607399936,"internally_reserved":0,"allocated":1691648,"data_stored":1076053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61036,"kb_used_data":29852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310804,"statfs":{"total":96636764160,"available":96574263296,"internally_reserved":0,"allocated":30568448,"data_stored":29852098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,5,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28684,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343156,"statfs":{"total":96636764160,"available":96607391744,"internally_reserved":0,"allocated":1699840,"data_stored":1083690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65196,"kb_used_data":29852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306644,"statfs":{"total":96636764160,"available":96570003456,"internally_reserved":0,"allocated":30568448,"data_stored":29852098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[2,3,4,5,6,7,8,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1691648,"data_stored":1076053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,4,6,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61024,"kb_used_data":29840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310816,"statfs":{"total":96636764160,"available":96574275584,"internally_reserved":0,"allocated":30556160,"data_stored":29844442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[1,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":108,"seq":463856467986,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86108,"kb_used_data":58124,"kb_used_omap":7,"kb_used_meta":27960,"kb_avail":94285732,"statfs":{"total":96636764160,"available":96548589568,"internally_reserved":0,"allocated":59518976,"data_stored":58745961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8160,"internal_metadata":28631072},"hb_peers":[1,2,3,5,6,7,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":19107840,"data_stored":19009285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22380544,"data_stored":22328092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":26083328,"data_stored":25968389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28868608,"data_stored":28768408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":57806848,"data_stored":57654565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3729,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":48463872,"data_stored":48296481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":35426304,"data_stored":35326473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T11:35:37.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T11:35:37.333 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T11:35:37.333 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T11:35:37.333 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T11:35:37.333 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.374 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T11:35:37.374 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T11:35:37.374 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.449 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T11:35:37.449 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-13T11:35:37.449 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.518 INFO:tasks.ceph.osd.8:Stopped 2024-09-13T11:35:37.519 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-13T11:35:37.519 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.591 INFO:tasks.ceph.osd.12:Stopped 2024-09-13T11:35:37.591 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T11:35:37.591 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.626 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T11:35:37.626 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T11:35:37.626 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.693 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T11:35:37.693 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-13T11:35:37.693 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.768 INFO:tasks.ceph.osd.9:Stopped 2024-09-13T11:35:37.768 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-13T11:35:37.768 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.840 INFO:tasks.ceph.osd.13:Stopped 2024-09-13T11:35:37.840 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T11:35:37.840 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.875 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T11:35:37.875 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T11:35:37.875 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:37.951 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T11:35:37.951 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-13T11:35:37.951 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.035 INFO:tasks.ceph.osd.10:Stopped 2024-09-13T11:35:38.035 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-13T11:35:38.035 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.077 INFO:tasks.ceph.osd.14:Stopped 2024-09-13T11:35:38.077 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T11:35:38.078 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.117 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T11:35:38.118 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T11:35:38.118 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.192 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T11:35:38.192 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-13T11:35:38.193 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.267 INFO:tasks.ceph.osd.11:Stopped 2024-09-13T11:35:38.267 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-13T11:35:38.267 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.343 INFO:tasks.ceph.osd.15:Stopped 2024-09-13T11:35:38.343 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T11:35:38.343 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T11:35:38.343 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.373 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T11:35:38.373 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T11:35:38.373 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.414 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T11:35:38.414 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T11:35:38.414 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T11:35:38.415 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.428 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T11:35:38.428 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T11:35:38.428 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.443 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T11:35:38.443 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T11:35:38.443 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T11:35:38.458 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T11:35:38.458 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T11:35:38.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-13T11:35:38.506 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:35:38.507 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T11:35:38.720 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:35:38.720 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T11:35:38.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:35:38.802 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-13T11:35:38.856 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:35:38.856 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-13T11:35:38.913 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:35:38.913 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T11:35:39.155 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:35:39.155 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T11:35:39.243 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:35:39.243 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-13T11:35:39.301 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:35:39.302 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-13T11:35:39.353 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:35:39.357 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T11:35:39.483 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:35:39.483 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T11:35:39.557 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:35:39.557 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-13T11:35:39.615 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:35:39.615 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-13T11:35:39.679 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:35:39.679 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T11:35:39.860 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:35:39.861 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T11:35:39.948 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:35:39.949 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-13T11:35:40.005 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:35:40.006 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-13T11:35:40.064 INFO:tasks.ceph:Archiving mon data... 2024-09-13T11:35:40.076 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/data/mon.a.tgz 2024-09-13T11:35:40.076 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-09-13T11:35:40.098 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.HomoimeorG 2024-09-13T11:35:40.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.HomoimeorG 2024-09-13T11:35:40.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.HomoimeorG 2024-09-13T11:35:40.365 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.HomoimeorG is 514KB 2024-09-13T11:35:40.466 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.HomoimeorG 2024-09-13T11:35:40.483 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/data/mon.b.tgz 2024-09-13T11:35:40.483 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-09-13T11:35:40.505 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.RNcGLPahl5 2024-09-13T11:35:40.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.RNcGLPahl5 2024-09-13T11:35:40.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.RNcGLPahl5 2024-09-13T11:35:40.790 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.RNcGLPahl5 is 526KB 2024-09-13T11:35:40.887 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.RNcGLPahl5 2024-09-13T11:35:40.902 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/data/mon.c.tgz 2024-09-13T11:35:40.902 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-13T11:35:40.922 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.9tVjTVYfa4 2024-09-13T11:35:40.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9tVjTVYfa4 2024-09-13T11:35:41.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.9tVjTVYfa4 2024-09-13T11:35:41.350 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.9tVjTVYfa4 is 526KB 2024-09-13T11:35:41.454 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.9tVjTVYfa4 2024-09-13T11:35:41.470 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T11:35:41.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:35:41.473 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:35:41.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:35:41.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T11:35:41.563 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:35:41.581 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:35:41.597 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:35:41.615 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T11:35:41.615 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T11:35:41.617 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi038/crash 2024-09-13T11:35:41.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:35:41.649 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi118/crash 2024-09-13T11:35:41.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:35:41.679 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi154/crash 2024-09-13T11:35:41.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:35:41.704 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi204/crash 2024-09-13T11:35:41.705 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T11:35:41.744 INFO:tasks.ceph:Compressing logs... 2024-09-13T11:35:41.745 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:35:41.748 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:35:41.750 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:35:41.752 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T11:35:41.774 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-13T11:35:41.774 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.37165.log 2024-09-13T11:35:41.774 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5/var/log/ceph/ceph-osd.10.log: --verbose -- /var/log/ceph/objectstore_tool.37925.log 2024-09-13T11:35:41.776 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/objectstore_tool.37165.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:35:41.776 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/objectstore_tool.37925.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39879.log 2024-09-13T11:35:41.776 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-13T11:35:41.776 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/objectstore_tool.39879.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39453.log 2024-09-13T11:35:41.776 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.36953.log 2024-09-13T11:35:41.779 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T11:35:41.779 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-13T11:35:41.779 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-13T11:35:41.779 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:35:41.779 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T11:35:41.779 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:35:41.779 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T11:35:41.782 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/objectstore_tool.39453.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T11:35:41.782 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/objectstore_tool.36953.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:35:41.782 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-13T11:35:41.782 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.2.log: 93.1% -- replaced with /var/log/ceph/objectstore_tool.39453.log.gz 2024-09-13T11:35:41.782 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:35:41.782 INFO:teuthology.orchestra.run.smithi118.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:35:41.789 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-13T11:35:41.789 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.14.log: 94.0% -- replaced with /var/log/ceph/objectstore_tool.36953.log.gz 2024-09-13T11:35:41.789 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:35:41.795 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39665.log 2024-09-13T11:35:41.796 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.log: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:35:41.796 INFO:teuthology.orchestra.run.smithi154.stderr: 93.4% -- replaced with /var/log/ceph/objectstore_tool.37925.log.gz 2024-09-13T11:35:41.800 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.36741.log 2024-09-13T11:35:41.801 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/objectstore_tool.39665.log: 94.1% -- replaced with /var/log/ceph/objectstore_tool.39879.log.gz 2024-09-13T11:35:41.805 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/objectstore_tool.36741.log: 94.1% -- replaced with /var/log/ceph/objectstore_tool.37165.log.gz 2024-09-13T11:35:41.806 INFO:teuthology.orchestra.run.smithi154.stderr: 93.3% -- replaced with /var/log/ceph/objectstore_tool.36741.log.gz 2024-09-13T11:35:41.811 INFO:teuthology.orchestra.run.smithi154.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.39665.log.gz 2024-09-13T11:35:41.830 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37981.log 2024-09-13T11:35:41.830 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32806.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.37981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32635.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37981.log.gz 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32806.log.gz 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33246.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32635.log.gz 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33086.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.33246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33046.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33246.log.gz 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.33086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32926.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33086.log.gz 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.33046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32675.log 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33046.log.gz 2024-09-13T11:35:41.831 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32926.log.gz 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33166.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32715.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32675.log.gz 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.33166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32759.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33166.log.gz 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32715.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32715.log.gz 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32966.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32759.log.gz 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38004.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33206.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32966.log.gz 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.0.38004.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-13T11:35:41.832 INFO:teuthology.orchestra.run.smithi204.stderr: 82.4% -- replaced with /var/log/ceph/ceph-client.0.38004.log.gz 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.33206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33206.log.gz 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33126.log 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32886.log 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.33126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33006.log 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33126.log.gz 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32886.log.gz 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32846.log 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.33006.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33006.log.gz 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.32846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32846.log.gz 2024-09-13T11:35:41.833 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.3.log: 92.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T11:35:41.882 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41287.log 2024-09-13T11:35:41.882 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T11:35:41.882 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55129.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41287.log.gz 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52777.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48628.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55129.log.gz 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52777.log.gz 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39316.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48628.log.gz 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53928.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39316.log.gz 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54631.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48332.log 2024-09-13T11:35:41.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53928.log.gz 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52441.log 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54631.log.gz 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55885.log 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48332.log.gz 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48885.log 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52441.log.gz 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55885.log.gz 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49586.log 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48885.log.gz 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47176.log 2024-09-13T11:35:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49586.log.gz 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52100.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47176.log.gz 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51173.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49948.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52100.log.gz 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.51173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53948.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51173.log.gz 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46295.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49948.log.gz 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53948.log.gz 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42643.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38699.log 2024-09-13T11:35:41.885 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46295.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53197.log 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42643.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43940.log 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38699.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55465.log 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53197.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56305.log 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43940.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39049.log 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55465.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38661.log 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56305.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39953.log 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39049.log.gz 2024-09-13T11:35:41.886 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47328.log 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38661.log.gz 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40037.log 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39953.log.gz 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53917.log 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47328.log.gz 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54584.log 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40037.log.gz 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54651.log 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53917.log.gz 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46254.log 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54584.log.gz 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49906.log 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54651.log.gz 2024-09-13T11:35:41.887 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46254.log.gz 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40158.log 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41678.log 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49906.log.gz 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48290.log 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40158.log.gz 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50302.log 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr: 21.0% -- replaced with /var/log/ceph/ceph-client.admin.41678.log.gz 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53932.log 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48290.log.gz 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54314.log 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50302.log.gz 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53932.log.gz 2024-09-13T11:35:41.888 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44325.log 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54314.log.gz 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53113.log 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52273.log 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44325.log.gz 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53113.log.gz 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46812.log 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52273.log.gz 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44152.log 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49378.log 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46812.log.gz 2024-09-13T11:35:41.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55843.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44152.log.gz 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33195.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49378.log.gz 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55843.log.gz 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52987.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33195.log.gz 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41522.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52987.log.gz 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38357.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46770.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41522.log.gz 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47815.log 2024-09-13T11:35:41.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46770.log.gz 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47815.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.47815.log.gz -- 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/ceph-client.admin.33399.log 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50436.log 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40388.log 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33399.log.gz 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55927.log 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr: 18.7% -- replaced with /var/log/ceph/ceph-client.admin.50436.log.gz 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54515.log 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40388.log.gz 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55927.log.gz 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56011.log 2024-09-13T11:35:41.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43827.log 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54515.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56011.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38164.log 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41428.log 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43827.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38164.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52357.log 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.41428.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39335.log 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49501.log 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52357.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39335.log.gz 2024-09-13T11:35:41.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55717.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44947.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49501.log.gz 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gz 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54574.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52525.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44947.log.gz 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49866.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54574.log.gz 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44051.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52525.log.gz 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55423.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49866.log.gz 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56179.log 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44051.log.gz 2024-09-13T11:35:41.893 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43084.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55423.log.gz 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52231.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56179.log.gz 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43084.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43084.log.gz 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52945.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52231.log.gz 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42033.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52945.log.gz 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53959.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39362.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42033.log.gz 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53959.log.gz 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-09-13T11:35:41.894 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39362.log.gz 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53896.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41993.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50798.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53896.log.gz 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41993.log.gz 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50798.log.gz 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46700.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53029.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51078.log 2024-09-13T11:35:41.895 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46700.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39176.log 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53029.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.51078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51078.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38273.log 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39176.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53930.log 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49648.log 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38273.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46924.log 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53930.log.gz 2024-09-13T11:35:41.896 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49648.log: /var/log/ceph/ceph-client.admin.52058.log 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49648.log.gz 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46924.log.gz 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39803.log 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46882.log 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52058.log.gz 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39803.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39803.log.gz -5 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53958.log 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43898.log 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46882.log.gz 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53958.log.gz 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40853.log 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52819.log 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43898.log.gz 2024-09-13T11:35:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40853.log.gz 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53449.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38683.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52819.log.gz 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53449.log.gz 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52609.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47500.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38683.log.gz 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52609.log.gz 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53924.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52184.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47500.log.gz 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53924.log.gz 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53323.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49543.log 2024-09-13T11:35:41.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52184.log.gz 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53323.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.53323.log.gz -5 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39743.log 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53533.log 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49543.log.gz 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39743.log.gz 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55045.log 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56723.log 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53533.log.gz 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55045.log.gz 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43390.log 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54621.log 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56723.log.gz 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43390.log.gz 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53071.log 2024-09-13T11:35:41.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48220.log 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54621.log.gz 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53071.log.gz 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55213.log 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54630.log 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48220.log.gz 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43410.log 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55213.log.gz 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42827.log 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54630.log.gz 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53281.log 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.43410.log.gz 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48394.log 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42827.log.gz 2024-09-13T11:35:41.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53281.log.gz 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55171.log 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43277.log 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48394.log.gz 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55171.log.gz 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42705.log 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43277.log: /var/log/ceph/ceph-client.admin.42572.log 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43277.log.gz 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42705.log.gz 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52735.log 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50301.log 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42572.log.gz 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45982.log 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52735.log.gz 2024-09-13T11:35:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53950.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr: 18.3% -- replaced with /var/log/ceph/ceph-client.admin.50301.log.gz 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.45982.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45982.log.gz -5 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46478.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54492.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53950.log.gz 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46478.log.gz 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55297.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39763.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54492.log.gz 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55297.log.gz 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40491.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40200.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39763.log.gz 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49308.log 2024-09-13T11:35:41.902 INFO:teuthology.orchestra.run.smithi038.stderr: 18.0% -- replaced with /var/log/ceph/ceph-client.admin.40491.log.gz 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39193.log 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40200.log.gz 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49308.log.gz 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40748.log 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49990.log 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39193.log.gz 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40748.log.gz 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39903.log 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38692.log 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49990.log.gz 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39903.log.gz 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45808.log 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38315.log 2024-09-13T11:35:41.903 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38692.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.45808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50910.log 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45808.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50648.log 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38315.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50910.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40242.log 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50648.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55339.log 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42106.log 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40242.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55339.log.gz 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53937.log 2024-09-13T11:35:41.904 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48547.log 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42106.log: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48058.log 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53937.log.gz 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48547.log.gz 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46366.log 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48058.log.gz 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38231.log 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46366.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.46366.log.gz -- 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/ceph-client.admin.52315.log 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38231.log.gz 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52315.log.gz 2024-09-13T11:35:41.905 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42767.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42176.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.42767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42767.log.gz 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38679.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38684.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42176.log.gz 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38048.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38679.log.gz 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45850.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38684.log.gz 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38048.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38048.log.gz 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43144.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.45850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45850.log.gz 2024-09-13T11:35:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52399.log 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43144.log.gz 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42704.log 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52399.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz -5 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph.tmp-client.admin.27093.log 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52142.log 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42704.log.gz 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.tmp-client.admin.27093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52903.log 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27093.log.gz 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55003.log 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52142.log.gz 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44555.log 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52903.log.gz 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55003.log.gz 2024-09-13T11:35:41.907 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.44301.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47772.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44555.log.gz 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/objectstore_tool.44301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47772.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47772.log.gz 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/ceph-client.admin.55591.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50503.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55591.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55591.log.gz 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/ceph-client.admin.50952.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50503.log.gz 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40894.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50952.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50952.log.gz 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42922.log 2024-09-13T11:35:41.908 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40894.log: 20.4% -- replaced with /var/log/ceph/ceph-client.admin.40894.log.gz 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56473.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42922.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42922.log.gz 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/ceph-client.admin.40078.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56473.log.gz 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.44659.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40078.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40078.log.gz 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53934.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/objectstore_tool.44659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55381.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53934.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53949.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53934.log.gz 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55381.log.gz 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39277.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53949.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:41.909 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46966.log 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39277.log: /var/log/ceph/ceph-mgr.y.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39277.log.gz 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/objectstore_tool.44301.log.gz 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49724.log 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46966.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39347.log 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46966.log.gz 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56263.log 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49724.log: /var/log/ceph/ceph-client.admin.39347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49724.log.gz 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39347.log.gz 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47928.log 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56263.log.gz 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38666.log 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38666.log: /var/log/ceph/ceph-client.admin.47928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38666.log.gz 18.3% -- replaced with /var/log/ceph/ceph-client.admin.47928.log.gz 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:41.910 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39368.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50394.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39368.log.gz 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47601.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50394.log.gz 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53239.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47601.log.gz 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43323.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53239.log.gz 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39356.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.43323.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56095.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43323.log.gz 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55255.log 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39356.log: /var/log/ceph/ceph-client.admin.56095.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39356.log.gz 2024-09-13T11:35:41.911 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56095.log.gz 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54570.log 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55255.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38694.log 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54570.log: /var/log/ceph/ceph-client.admin.40284.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54570.log.gz 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54607.log 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38694.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55507.log 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38694.log.gz 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45738.log 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54607.log.gz 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55507.log.gz 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54545.log 2024-09-13T11:35:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.45738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53956.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45738.log.gz 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50840.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54545.log.gz 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53956.log.gz 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56053.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50840.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47732.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50840.log.gz 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52861.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56053.log.gz 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47732.log.gz 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50992.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52861.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54606.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52861.log.gz 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.50992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46386.log 2024-09-13T11:35:41.913 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50992.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54606.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47370.log 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46386.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43492.log 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46386.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55969.log 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47370.log: 18.3%/var/log/ceph/ceph-client.admin.43492.log: -- replaced with /var/log/ceph/ceph-client.admin.47370.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43492.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56641.log 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55969.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.38691.log 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55969.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46042.log 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56641.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38691.log.gz 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47840.log 2024-09-13T11:35:41.914 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.46042.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54495.log 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46042.log.gz 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49398.log 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47840.log: /var/log/ceph/ceph-client.admin.54495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47840.log.gz 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54495.log.gz 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48820.log 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49398.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32927.log 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48820.log: /var/log/ceph/ceph-client.admin.32927.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32927.log.gz 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48820.log.gz 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53947.log 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47258.log 2024-09-13T11:35:41.915 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53947.log.gz 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54645.log 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.47258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47258.log.gz 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56557.log 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.54645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54645.log.gz 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55801.log 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56557.log.gz 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49086.log 2024-09-13T11:35:41.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55801.log.gz 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38682.log 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49086.log.gz 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39185.log 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38682.log.gz 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53155.log 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39185.log.gz 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53155.log.gz 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48973.log 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.40956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48973.log.gz 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52567.log 2024-09-13T11:35:41.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56389.log 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52567.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39355.log 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56389.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr: 94.2% -- replaced with /var/log/ceph/objectstore_tool.44659.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56347.log 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39355.log: /var/log/ceph/ceph-client.admin.55633.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39355.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55549.log 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48374.log 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56347.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55549.log.gz 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-09-13T11:35:41.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44691.log 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39372.log: /var/log/ceph/ceph-client.admin.48374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48374.log.gz 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55759.log 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49460.log 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.44691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44691.log.gz 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55759.log.gz 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53575.log 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55675.log: /var/log/ceph/ceph-client.admin.49460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49460.log.gz 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48458.log 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39825.log 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53575.log.gz 2024-09-13T11:35:41.949 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.48458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48458.log.gz 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56683.log 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49616.log 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56683.log: /var/log/ceph/ceph-client.admin.39825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39825.log.gz 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56683.log.gz 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56137.log 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56221.log/var/log/ceph/ceph-client.admin.49616.log: 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49616.log.gz 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56599.log 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56137.log: /var/log/ceph/ceph-client.admin.56221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56137.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56221.log.gz 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52651.log 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56599.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54591.log 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56599.log.gz 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53927.log 2024-09-13T11:35:41.950 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52651.log: /var/log/ceph/ceph-client.admin.54591.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52651.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54591.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52483.log 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.53927.log: -- /var/log/ceph/ceph-client.admin.39995.log 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53927.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0%/var/log/ceph/ceph-client.admin.39995.log: -- replaced with /var/log/ceph/ceph-client.admin.52483.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39995.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38123.log 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.41030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49782.log 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr: 19.6% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55087.log 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38123.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49782.log.gz 2024-09-13T11:35:41.951 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50071.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.55087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55087.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38651.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38673.log: /var/log/ceph/ceph-client.admin.50071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50071.log.gz 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53407.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51034.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38651.log.gz 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53407.log.gz 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56431.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45920.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56431.log: /var/log/ceph/ceph-client.admin.51034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51034.log.gz 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56431.log.gz 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53491.log 2024-09-13T11:35:41.952 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42319.log 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.45920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45920.log.gz 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53491.log.gz 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56515.log 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39233.log 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.56515.log: /var/log/ceph/ceph-client.admin.42319.log: 19.1% -- replaced with /var/log/ceph/ceph-client.admin.42319.log.gz 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56515.log.gz 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48905.log 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.39233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39233.log.gz 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52693.log 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42510.log 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.52693.log: /var/log/ceph/ceph-client.admin.48905.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52693.log.gz -- replaced with /var/log/ceph/ceph-client.admin.48905.log.gz 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:41.953 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53365.log 2024-09-13T11:35:41.954 INFO:teuthology.orchestra.run.smithi118.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T11:35:41.954 INFO:teuthology.orchestra.run.smithi154.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T11:35:41.955 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49016.log 2024-09-13T11:35:41.955 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.42510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42510.log.gz 2024-09-13T11:35:41.955 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.53365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53365.log.gz 2024-09-13T11:35:41.956 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.49016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49016.log.gz 2024-09-13T11:35:41.956 INFO:teuthology.orchestra.run.smithi038.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-13T11:35:42.110 INFO:teuthology.orchestra.run.smithi118.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T11:35:42.203 INFO:teuthology.orchestra.run.smithi038.stderr: 90.5% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-13T11:35:42.277 INFO:teuthology.orchestra.run.smithi038.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T11:35:42.320 INFO:teuthology.orchestra.run.smithi154.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-13T11:35:42.446 INFO:teuthology.orchestra.run.smithi038.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T11:35:42.454 INFO:teuthology.orchestra.run.smithi204.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-13T11:35:42.473 INFO:teuthology.orchestra.run.smithi038.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-13T11:35:42.491 INFO:teuthology.orchestra.run.smithi154.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T11:35:42.526 INFO:teuthology.orchestra.run.smithi118.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-13T11:35:42.555 INFO:teuthology.orchestra.run.smithi118.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T11:35:42.570 INFO:teuthology.orchestra.run.smithi154.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-13T11:35:42.642 INFO:teuthology.orchestra.run.smithi154.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-13T11:35:42.644 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-13T11:35:42.644 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m0.879s 2024-09-13T11:35:42.644 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m3.072s 2024-09-13T11:35:42.644 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.131s 2024-09-13T11:35:42.786 INFO:teuthology.orchestra.run.smithi204.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T11:35:42.799 INFO:teuthology.orchestra.run.smithi038.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T11:35:42.801 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-09-13T11:35:42.801 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m1.038s 2024-09-13T11:35:42.802 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m3.268s 2024-09-13T11:35:42.802 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.303s 2024-09-13T11:35:42.868 INFO:teuthology.orchestra.run.smithi118.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-13T11:35:42.871 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-09-13T11:35:42.871 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m1.105s 2024-09-13T11:35:42.871 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m3.021s 2024-09-13T11:35:42.871 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.132s 2024-09-13T11:35:42.878 INFO:teuthology.orchestra.run.smithi204.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-13T11:35:43.152 INFO:teuthology.orchestra.run.smithi204.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-13T11:35:43.153 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-09-13T11:35:43.153 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m1.350s 2024-09-13T11:35:43.154 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m3.862s 2024-09-13T11:35:43.154 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.171s 2024-09-13T11:35:43.154 INFO:tasks.ceph:Archiving logs... 2024-09-13T11:35:43.154 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi038/log 2024-09-13T11:35:43.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:35:43.965 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi118/log 2024-09-13T11:35:43.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:35:44.560 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi154/log 2024-09-13T11:35:44.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:35:45.175 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi204/log 2024-09-13T11:35:45.176 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T11:35:45.885 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T11:35:45.895 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T11:35:45.895 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:35:45.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:35:45.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:35:45.907 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T11:35:46.061 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T11:35:46.073 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T11:35:46.073 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:35:46.076 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:35:46.079 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:35:46.082 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T11:35:46.092 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:35:46.093 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:35:46.097 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:35:46.098 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-09-13T11:35:46.098 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 -67us[ -66us] +/- 39ms 2024-09-13T11:35:46.098 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 +552us[ +552us] +/- 52ms 2024-09-13T11:35:46.098 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 -586us[ -585us] +/- 70ms 2024-09-13T11:35:46.098 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:35:46.099 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:35:46.101 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:35:46.101 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-09-13T11:35:46.101 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 -59us[ -55us] +/- 39ms 2024-09-13T11:35:46.106 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 +546us[ +550us] +/- 52ms 2024-09-13T11:35:46.106 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -578us[ -578us] +/- 70ms 2024-09-13T11:35:46.106 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:35:46.107 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 -48us[ -34us] +/- 39ms 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 +568us[ +568us] +/- 52ms 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 -564us[ -549us] +/- 70ms 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-09-13T11:35:46.109 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 -71us[ -69us] +/- 39ms 2024-09-13T11:35:46.110 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 +545us[ +547us] +/- 52ms 2024-09-13T11:35:46.110 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 -583us[ -581us] +/- 70ms 2024-09-13T11:35:46.110 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T11:35:46.110 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T11:35:46.119 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T11:35:46.120 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T11:35:46.128 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T11:35:46.217 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T11:35:46.290 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T11:35:46.361 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T11:35:46.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:35:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:46.465 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:35:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:46.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:35:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:46.530 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T11:35:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:46.561 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T11:35:46.572 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T11:35:46.581 INFO:teuthology.task.internal:Duration was 689.105665 seconds 2024-09-13T11:35:46.582 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T11:35:46.591 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T11:35:46.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:35:46.593 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:35:46.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:35:46.599 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T11:35:46.635 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:35:46.639 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:35:46.639 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:35:46.642 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T11:35:47.017 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T11:35:47.017 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-09-13T11:35:47.017 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:35:47.042 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-09-13T11:35:47.042 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:35:47.065 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-09-13T11:35:47.066 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:35:47.093 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-09-13T11:35:47.094 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T11:35:47.129 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T11:35:47.130 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:35:47.133 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:35:47.135 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:35:47.137 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T11:35:47.199 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T11:35:47.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:35:47.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:35:47.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:35:47.209 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T11:35:47.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T11:35:47.359 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T11:35:47.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:35:47.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:35:47.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:35:47.436 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T11:35:47.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T11:35:47.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.479 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.495 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-09-13T11:35:47.497 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-09-13T11:35:47.500 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-09-13T11:35:47.525 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-09-13T11:35:47.561 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:47.576 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:47.592 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:47.607 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T11:35:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T11:35:47.622 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T11:35:47.632 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T11:35:47.633 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi038 2024-09-13T11:35:47.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:35:47.669 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi118 2024-09-13T11:35:47.670 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:35:47.705 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi154 2024-09-13T11:35:47.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:35:47.745 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:37:56-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903421/remote/smithi204 2024-09-13T11:35:47.745 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T11:35:47.780 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T11:35:47.781 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:35:47.783 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:35:47.786 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:35:47.789 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T11:35:47.830 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T11:35:47.841 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T11:35:47.841 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T11:35:47.851 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T11:35:47.851 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:35:47.854 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:35:47.856 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:35:47.858 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T11:35:47.867 INFO:teuthology.orchestra.run.smithi038.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:35 /home/ubuntu/cephtest 2024-09-13T11:35:47.868 INFO:teuthology.orchestra.run.smithi118.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:35 /home/ubuntu/cephtest 2024-09-13T11:35:47.871 INFO:teuthology.orchestra.run.smithi154.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:35 /home/ubuntu/cephtest 2024-09-13T11:35:47.885 INFO:teuthology.orchestra.run.smithi204.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 11:35 /home/ubuntu/cephtest 2024-09-13T11:35:47.886 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T11:35:47.898 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T11:35:47.922 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/pggrow_host thrashosds-health workloads/dedup-io-mixed} duration: 689.1056652069092 flavor: default owner: scheduled_akupczyk@teuthology success: true 2024-09-13T11:35:47.923 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T11:35:48.023 INFO:teuthology.run:pass